Skip to content

fix(proxy): crash safety, do not execute writes abandoned by their client - #35

Merged
sanlega merged 1 commit into
mainfrom
fix/34-abandoned-writes
Nov 13, 2026
Merged

sanlega merged 1 commit into
mainfrom
fix/34-abandoned-writes

Conversation

@sanlega

@sanlega sanlega commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Closes #34

What and why

Feedback: "Have you tested recovery after stopping the emulator mid-write, or mainly clean shutdowns so far?" So far only clean restarts and docker kill between applies had been tested. This PR tests crashes in the middle of terraform apply, fixes the bug it found and keeps the tests running.

Bug: writes wait for the write lock while keepstack restores the emulator (or behind a slow write). If the client gave up meanwhile (an SDK attempt timing out, closing the connection and retrying), keepstack still executed the abandoned request when the lock freed. The resource was created and journaled, the client's retry got BucketAlreadyExists / ResourceInUseException, and the resource never reached the Terraform state. Seen in 2 of 8 runs where the emulator was killed mid-apply.

Fix: App.handle() gets a client_gone check and runs it, for writes, after the request is allowed through (ready + lock) and before it is journaled or forwarded. The HTTP handler implements it with socket_closed() (select + recv(MSG_PEEK): readable with 0 bytes = the peer closed; pending bytes = still there). An abandoned request gets no answer, is not journaled and is logged as client gave up waiting; not executing <op>. Reads are answered anyway.

Chaos suite: scripts/chaos.py applies the quickstart example and, at a random moment, kills the emulator (docker kill), keepstack (kill -9, restarted while Terraform retries) or everything at once (power cut), then requires the re-apply to converge and the plan to stay clean after another clean restart. New nightly workflow chaos.yml (+ manual dispatch). Documented in docs/how-it-works.md#crash-safety.

How it was tested

  • Deterministic reproduction (200 queues in the journal, docker kill, CreateQueue with a 1 s client timeout while it restores), now tests/test_e2e_crash.py: fails without the fix (the abandoned CreateQueue was executed after its client left), passes with it.
  • Unit: socket_closed (open / pipelined bytes / closed) and handle skipping an abandoned write (no journal row, no forward) while still answering reads. test_unit: 17 OK; all unit tests 118 OK.
  • Chaos, 12 iterations with the fix (seed 21):
Mode Runs First apply Converged
emulator killed 4 finished by itself 4/4 4/4
keepstack killed and restarted 4 finished by itself 4/4 4/4
power cut 4 killed (as Terraform is) re-apply converged 4/4

One power-cut run then failed the extra clean restart: a Route 53 zone ID made only of letters (ZSOVGDZPSPRJKSM) is not recognized as generated, so it is not translated. Unrelated to crashes; tracked separately in #36.

A write that waited for the restore of the emulator (or behind a slow
write) was executed even if its client had timed out and retried, creating
the resource behind Terraform's back. Check the connection first.

Adds scripts/chaos.py (kill the emulator, keepstack or everything
mid-apply) with a nightly workflow, and an e2e regression test.

Refs #34
@sanlega sanlega added this to the v0.4.0 milestone Nov 13, 2026
@sanlega sanlega added type: bug Something is broken area: proxy Journal, replay, ID translation labels Nov 13, 2026
@github-actions

Copy link
Copy Markdown

keepstack validate: examples/quickstart ✅ passed

Step Result
init ✅ ok
plan ✅ ok
apply ✅ ok
idempotency ✅ ok
smoke ✅ ok

Plan: 0 to add, 0 to change, 0 to replace, 0 to destroy

keepstack · 15.5 s · AWS Emulator, no AWS account used

@sanlega
sanlega merged commit c99e219 into main Nov 13, 2026
12 checks passed
@sanlega
sanlega deleted the fix/34-abandoned-writes branch November 13, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: proxy Journal, replay, ID translation type: bug Something is broken

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash safety: writes abandoned by the client while keepstack restores are still executed

1 participant