Repository navigation
test(chaos): power cuts can leave resources Terraform lost, as on AWS - #40
Merged
Merged
Conversation
In power mode, re-apply errors that only say "already exists" are resources Terraform created but never saved. The iteration passes if keepstack keeps exactly the same state across another restart. Refs #39
keepstack validate:
|
| Step | Result |
|---|---|
| init | ✅ ok |
| plan | ✅ ok |
| apply | ✅ ok |
| idempotency | ✅ ok |
| smoke | ✅ ok |
Plan: 0 to add, 0 to change, 0 to replace, 0 to destroy
keepstack · 16.5 s · AWS Emulator, no AWS account used
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39
What and why
First chaos run on Linux: 5/6. The failure was a power cut (Terraform, keepstack and the emulator killed together) while Terraform waited for a DynamoDB table to become ACTIVE: the emulator had created it, keepstack kept it (
13 calls replayed, 0 failed), but Terraform never saved it, so the re-apply saidTable already exists. That is what Terraform does on real AWS too; the suite's expectation was wrong (it passed on Windows by timing).scripts/chaos.py: inpowermode, if every re-apply error is an "already exists" error, the resources are reported asorphaned_by_terraformand the iteration passes only if keepstack keeps exactly the same state across another clean restart (identical planned changes before and after). Any other error is still a failure.docs/how-it-works.md#crash-safety: the power-cut row now says what really happens and how to recover (terraform import).tests/test_chaos.py:orphans()with the real Linux output, a mixed real error, and the planned-address parser.How it was tested
tests.test_chaos: 3 OK.aws_dynamodb_table.items;aws_dynamodb_table.items,aws_s3_bucket.data) and an identical plan before and after another restart.