Repository navigation
fix(stovepipe): admit before counting the build slot - #974
Conversation
|
Design question (Supervisor Fable): there is one residual path that leaves Two ways to handle it:
#966 closes it exactly by keying the slot on the request ID. I have left the code at option 1 pending your call; say "add the re-check" and I will have the subagent add it. |
Process claimed a build slot before moving the request to processing, so a crash between the two writes left in_flight_count raised with no owner and every redelivery held behind its own claim forever. Write the request transition first and increment the count after it, retrying conflicts without re-gating, and drop the unwind releases. A failure between the writes now leaves the count low by one, which the release paths' zero clamp reconverges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
9a60183 to
402380f
Compare
I am not sure which solution is better, I did not want the complexity of #966 so I've decided that those failures are rare enough for the gate to be made fail-open, and it is way less code to maintain. If you think to reconsider towards #966 I will be ok with it. There are trade offs to both. |
Why?
processadmits the latest head with two ordered CAS writes on two rows: it increments the anonymousqueue.in_flight_countfirst, then moves the request accepted→processing. A crash between them leaves the count raised with no owner. Holds never consume retry budget and the DLQ reconciler releases onlyprocessingrequests, so the queue wedges permanently.max_concurrent = 1; request R is accepted and is the latest head.processclaims the slot (in_flight_count = 1) and crashes before writingprocessing.accepted, finds the gate closed by its own claim, and holds.What?
admitLatestHeadgates and derives the strategy as before, then CASes the request accepted→processing first and incrementsin_flight_countsecond. A version conflict on the increment reloads and retries without re-gating or coalescing, because the admit is already durable. A lost request CAS acks without touching the counter.process'sreleaseBuildSlothelper and both unwind call sites.buildsignaland DLQ release paths brings the count back to the true value as the queue drains. The count can go high only if a delivery stalls between the two writes past the visibility timeout and wakes after its redelivery has finished the validation; the RFC records this residual and how to narrow or close it.doc/rfc/stovepipe/steps/process.md: step 7 reordered; the redelivery andin_flight_count integritysections now describe the fail-open order. Twodlq.go/buildsignal.godoc comments that described the old claim-first order are corrected (comments only).in_flight_count.Alternative to #966, which keys slots by request ID in a new
in_flight_request_idscolumn and is the exact fix; this PR needs no schema change. Merge one or the other, not both.Test Plan
make fmt && make lint && make check-tidy && make check-gazelle && make check-mocks./tool/bazel test //stovepipe/... //service/stovepipe/... --test_output=errors./tool/bazel build //test/e2e/stovepipe:go_default_test //test/integration/stovepipe/...Audit: SP-OCT-003 and SP-OCT-005 in doc/audit/stovepipe-fable-v2.md (Sol v2 #1/#2, Opus v2 #1/#8).
🤖 Generated with Claude Code