Repository navigation
fix(proxy): crash safety, do not execute writes abandoned by their client - #35
Merged
Merged
Conversation
A write that waited for the restore of the emulator (or behind a slow write) was executed even if its client had timed out and retried, creating the resource behind Terraform's back. Check the connection first. Adds scripts/chaos.py (kill the emulator, keepstack or everything mid-apply) with a nightly workflow, and an e2e regression test. Refs #34
keepstack validate:
|
| Step | Result |
|---|---|
| init | ✅ ok |
| plan | ✅ ok |
| apply | ✅ ok |
| idempotency | ✅ ok |
| smoke | ✅ ok |
Plan: 0 to add, 0 to change, 0 to replace, 0 to destroy
keepstack · 15.5 s · AWS Emulator, no AWS account used
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #34
What and why
Feedback: "Have you tested recovery after stopping the emulator mid-write, or mainly clean shutdowns so far?" So far only clean restarts and
docker killbetween applies had been tested. This PR tests crashes in the middle ofterraform apply, fixes the bug it found and keeps the tests running.Bug: writes wait for the write lock while keepstack restores the emulator (or behind a slow write). If the client gave up meanwhile (an SDK attempt timing out, closing the connection and retrying), keepstack still executed the abandoned request when the lock freed. The resource was created and journaled, the client's retry got
BucketAlreadyExists/ResourceInUseException, and the resource never reached the Terraform state. Seen in 2 of 8 runs where the emulator was killed mid-apply.Fix:
App.handle()gets aclient_gonecheck and runs it, for writes, after the request is allowed through (ready + lock) and before it is journaled or forwarded. The HTTP handler implements it withsocket_closed()(select+recv(MSG_PEEK): readable with 0 bytes = the peer closed; pending bytes = still there). An abandoned request gets no answer, is not journaled and is logged asclient gave up waiting; not executing <op>. Reads are answered anyway.Chaos suite:
scripts/chaos.pyapplies the quickstart example and, at a random moment, kills the emulator (docker kill), keepstack (kill -9, restarted while Terraform retries) or everything at once (power cut), then requires the re-apply to converge and the plan to stay clean after another clean restart. New nightly workflowchaos.yml(+ manual dispatch). Documented indocs/how-it-works.md#crash-safety.How it was tested
docker kill,CreateQueuewith a 1 s client timeout while it restores), nowtests/test_e2e_crash.py: fails without the fix (the abandoned CreateQueue was executed after its client left), passes with it.socket_closed(open / pipelined bytes / closed) andhandleskipping an abandoned write (no journal row, no forward) while still answering reads.test_unit: 17 OK; all unit tests 118 OK.One power-cut run then failed the extra clean restart: a Route 53 zone ID made only of letters (
ZSOVGDZPSPRJKSM) is not recognized as generated, so it is not translated. Unrelated to crashes; tracked separately in #36.