Repository navigation
fix(stovepipe): fail validations that exceed max_validation_ms - #967
sbalabanov wants to merge 1 commit into
Conversation
655b844 to
cfb56bd
Compare
buildsignal held a nonterminal build unconditionally, and holds do not consume the retry budget, so a runner that kept answering accepted or running was re-polled forever. The request never dead-lettered and, at MaxConcurrent 1, kept the queue's only build slot. Record Build.CreatedAt and add QueueConfig.MaxValidationMs (two hours in the reference config; zero disables it). When a poll leaves the build nonterminal past the deadline, buildsignal runs the normal terminal path and marks the request failed with reason validation_timeout. The timed-out build established nothing: it keeps its polled status, is not recorded as the terminal build, and gets no build_finished entry or terminal analysis. A retry re-derives the same timeout from the poll, and finishRequest is a no-op once the outcome is stored. Existing MySQL deployments must add created_at to build before deploying. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cfb56bd to
37e2795
Compare
|
Two things for the reviewer, neither blocks CI: Default value needs a human call. Merge order. This PR is stacked on #965. It also touches |
|
Closing without merging. If a bound turns out to be necessary, it will be a single deadline per whole request (admitted to terminal), not a per-build or per-external-system timeout. The build-runner poll is the only unbounded wait today, so one request-level deadline with one setting per queue covers it and any future asynchronous stage. Until then a too-old nonterminal build stays an operational alert, as the buildsignal RFC states. |
Why?
buildsignal holds a nonterminal build on every poll, and holds do not consume the retry budget, so a runner that keeps answering
acceptedorrunningis re-polled forever. Nothing dead-letters and nothing releases the build slot. Repro:MaxConcurrent1.runningon every poll, so the buildsignal message holds and redelivers indefinitely.What?
BuildgainsCreatedAt, which build sets at creation and MySQL persists in a newbuild.created_atcolumn.QueueConfiggainsMaxValidationMs(zero disables the deadline). The reference default store sets it to two hours. When a poll leaves the build nonterminal past the deadline, buildsignal runs the same terminal sequence as a polled failure and marks the requestfailedwith reasonvalidation_timeout(stored on the request by #965). The timed-out build established nothing: it keeps its last polled status, the request'sTerminalBuildIDstays empty, and there is nobuild_finishedentry or terminal analysis publish. Existing MySQL deployments must addcreated_at BIGINT NOT NULL DEFAULT 0tobuildfirst.Convergence: a retry after any failure in the timeout sequence polls again, finds the build still nonterminal and still past the deadline, and takes the same path with the same reason;
finishRequestis a no-op once the outcome landed. If the build reached a real terminal status in the meantime and the request has no outcome yet, that real verdict is recorded instead. Nothing lists nonterminal builds and the request is terminal, so nothing reopens.Test Plan
make fmt && make lint && make check-tidy && make check-gazelle && make check-mocks./tool/bazel test //stovepipe/... //service/stovepipe/... --test_output=errors && ./tool/bazel test //test/integration/stovepipe/extension/storage/mysql:go_default_test --test_output=errorsStack
Audit: SP-OCT-002 in doc/audit/stovepipe-fable-v2.md (Sol v2 #12, Opus v2 #4).
🤖 Generated with Claude Code