Skip to content

test(chaos): power cuts can leave resources Terraform lost, as on AWS - #40

Merged
sanlega merged 1 commit into
mainfrom
fix/39-chaos-power-orphans
Nov 13, 2026
Merged

sanlega merged 1 commit into
mainfrom
fix/39-chaos-power-orphans

Conversation

@sanlega

@sanlega sanlega commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Closes #39

What and why

First chaos run on Linux: 5/6. The failure was a power cut (Terraform, keepstack and the emulator killed together) while Terraform waited for a DynamoDB table to become ACTIVE: the emulator had created it, keepstack kept it (13 calls replayed, 0 failed), but Terraform never saved it, so the re-apply said Table already exists. That is what Terraform does on real AWS too; the suite's expectation was wrong (it passed on Windows by timing).

  • scripts/chaos.py: in power mode, if every re-apply error is an "already exists" error, the resources are reported as orphaned_by_terraform and the iteration passes only if keepstack keeps exactly the same state across another clean restart (identical planned changes before and after). Any other error is still a failure.
  • docs/how-it-works.md#crash-safety: the power-cut row now says what really happens and how to recover (terraform import).
  • tests/test_chaos.py: orphans() with the real Linux output, a mixed real error, and the planned-address parser.

How it was tested

  • tests.test_chaos: 3 OK.
  • Chaos on Linux from this branch (run 43342686266): 9/9. Emulator killed 3/3 and keepstack killed 3/3 finished by themselves; power cuts 3/3, two of them with resources Terraform lost (aws_dynamodb_table.items; aws_dynamodb_table.items, aws_s3_bucket.data) and an identical plan before and after another restart.

In power mode, re-apply errors that only say "already exists" are
resources Terraform created but never saved. The iteration passes if
keepstack keeps exactly the same state across another restart.

Refs #39
@sanlega sanlega added this to the v0.4.0 milestone Nov 13, 2026
@sanlega sanlega added the type: bug Something is broken label Nov 13, 2026
@github-actions

Copy link
Copy Markdown

keepstack validate: examples/quickstart ✅ passed

Step Result
init ✅ ok
plan ✅ ok
apply ✅ ok
idempotency ✅ ok
smoke ✅ ok

Plan: 0 to add, 0 to change, 0 to replace, 0 to destroy

keepstack · 16.5 s · AWS Emulator, no AWS account used

@sanlega
sanlega merged commit d3576fc into main Nov 13, 2026
13 checks passed
@sanlega
sanlega deleted the fix/39-chaos-power-orphans branch November 13, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: bug Something is broken

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chaos: a power cut can leave resources Terraform lost, as on AWS (suite expectation)

1 participant