A server that stops answering is ended, so the platform restarts it - #494
Merged
Merged
Conversation
TWO FAILURES WITH ONE SYMPTOM: an empty branch and a turn that reads as a model failure. 80 of 101 stuck issues on 2026-09-20 had exactly that shape. THE CHECKOUT. Measured in the running deployment's log: error: Your local changes to the following files would be overwritten by checkout: error: unable to create file specs/port/tools/gft_deep_demo.t27: Permission denied $WORKSPACE_DIR was owned by the bee, so the one-time ownership walk was skipped, while files underneath it were not - left by a root-run git from an older image. `find ! -user -print -quit` stops at the FIRST wrong file, so the healthy case costs one stat and the 45 GB walk that once outlasted the 300 s healthcheck cannot come back. The repair walks the checkout only, never the worktrees beside it, and changes only what is wrong. THE FETCH. Measured at concurrency four on one key against integrate.api.nvidia.com: two answers 200, one 503, and one socket that never answered at all. The retry wrapper handled 429 and 5xx and the 200-carrying-an- error case, and could not see the fourth - there is no response to branch on, so it reached the agent loop as a terminal error and ended the turn. A throw is now retried on the same backoff. An abort the CALLER asked for is re-thrown at once: retrying a cancelled request outlives the thing that cancelled it. bun test apps/server/src/lib/overload-retry-fetch.test.ts 10 pass, 0 fail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every tick for six minutes chose an issue and then refused to start it:
Queen tick chose an issue but the container cannot carry another bee
issue=4438 resource="disk"
Twenty lanes open, 684 candidates waiting, and zero bees - because the volume
had filled with bee worktrees nobody could use. An earlier reading of the same
volume found 41 of them holding 45 GB and three million inodes.
At entrypoint time no bee is running - this process is what starts the server
that starts them - so every directory under .worktrees/ belongs to a container
that is already gone. `git worktree prune` alone does not do it: it drops the
admin records for directories already removed, and these are still there.
Free space is printed before and after, so the next reader sees what it bought.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A refusal costs a provider call and returns nothing, and the provider is the ceiling. Measured on the running deployment 2026-09-20 in one 400-line window: 18 of 19 filesystem tool failures were this refusal, on files of 159 and 500 lines - ordinary specs - while the same endpoint answered `Service temporarily overloaded` 76 times in that same window. Every one of those refusals spent a round trip to be told to ask again. filesystem_read now returns the lines that fit under the character limit and names the exact offset to continue from, which is what the caller would have asked for on its second call. Room is kept for that note, so the answer can always carry one. A single line longer than the whole budget still throws - there is nothing to hand back - and now says so in those words, pointing at filesystem_grep. bun test apps/server/tests/tools/filesystem/read.test.ts 17 pass, 0 fail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…utes `runRound` awaits the review sweep before it reaps and before it dispatches, so the sweep's deadline is how long the swarm is willing to hand out no work at all. It was an absolute four minutes against a sixty-second tick - four rounds of silence, with the lease heartbeat reporting health throughout. Measured 2026-09-20: the swarm went from 8.5 to 80 dispatches an hour when the worker model changed, and the review sweep - three a round, unchanged since it was written - fell behind, sixteen to twenty unreviewed all afternoon. Raising the COUNT to eight reviews and six measurements killed the container five minutes later, because a count bounds worktrees and not time: each measurement cuts a temporary worktree and runs up to twenty commands. So the bound is time, and a fraction of the tick: 45 s of a 60 s round, floored at 20 s so a very short tick cannot make review impossible, and ceilinged at the old four minutes for a deployment whose tick is minutes long. The count can now be generous because the clock is what protects the dispatcher. The deadline applies to REVIEWS as well as measurements. A review is a provider call and can sit on its own timeout; bounding only the measurements left the dispatcher waiting on the half that was never counted. bun test queen-tick-sweep queen-review-unjudged queen-adversarial-review queen-criteria-run queend-choose 92 pass, 0 fail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The emergency sweep keeps any worktree whose branch carries commits the base does not, because the container holds no push credential and such a commit would live in exactly one place. That is right, and it stops being true the moment the branch is on origin at the same commit. Measured 2026-09-20: the entrypoint found TWENTY-ONE worktrees at boot, every one of them kept by that branch of the check. The volume filled, the process was killed, the entrypoint cleared them, the swarm ran for a few minutes and it happened again - six restarts in one afternoon, at four lanes as readily as at twenty, because the leak is per FINISHED bee and not per running one. So the sweep now reads the remote. A branch on origin at the same SHA is published, and its worktree is disk. Reading the remote needs no push credential; if it cannot be read the tree stays, because unreachable is not published for the same reason unreadable is not clean. bun test apps/server/tests/api/queen-dispatch.test.ts 96 pass, 0 fail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…estarts it Measured 2026-09-20/21: the agent server twice stopped answering HTTP WITHOUT EXITING. The edge said `Application failed to respond`, and Railway's restart policy never fired, because a restart policy fires on an exit and there was none. The first time it stayed down for nine hours. Raising restartPolicyMaxRetries did not help the second time, for the same reason. An outside watchdog can redeploy, but only with a platform token. A process inside the container needs nothing but loopback. So the entrypoint no longer `exec`s the server. It starts it, and a second process asks `/health` on 127.0.0.1 every LIVENESS_INTERVAL (30 s) after a LIVENESS_GRACE (240 s) boot allowance. LIVENESS_FAILS (4) misses in a row and the server gets SIGTERM, then SIGKILL; the entrypoint exits non-zero and ON_FAILURE brings the container back - through the boot clean-up that frees the disk. One healthy answer resets the count. python3 does the asking because the image declares it; there is no curl here. Proven before shipping on a real HTTP server frozen with SIGSTOP - alive, not answering, the production failure exactly: [liveness] /health did not answer (1 of 3) [liveness] /health did not answer (2 of 3) [liveness] /health did not answer (3 of 3) [liveness] the server stopped answering; ending it so the platform restarts the container [liveness] the server exited (143); exiting so the platform restarts the container Also restores restartPolicyMaxRetries 10000 in railway.json: it was deployed from an uncommitted working tree and lost on the next branch switch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
❌ Tests failed — 11/2436 failed
Failed tests
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agent server twice stopped answering HTTP without exiting (
Application failed to respond). Railway's restart policy fires on an exit, so it never fired — the first time the swarm stayed down nine hours.The entrypoint now starts the server and runs a second process that asks
/healthon loopback every 30 s after a 240 s boot allowance. Four misses in a row: SIGTERM, then SIGKILL, and the entrypoint exits non-zero soON_FAILURErestarts the container — through the boot clean-up that frees the disk. No platform token needed.Proven on a real HTTP server frozen with SIGSTOP (alive, not answering):
This branch stacks on #491, #492 and #493, which are what is running in production. It also restores
restartPolicyMaxRetries: 10000, which had been deployed from an uncommitted tree and lost.🤖 Generated with Claude Code