A bee that cannot write the checkout, and a fetch that threw - #491
Conversation
TWO FAILURES WITH ONE SYMPTOM: an empty branch and a turn that reads as a model failure. 80 of 101 stuck issues on 2026-09-20 had exactly that shape. THE CHECKOUT. Measured in the running deployment's log: error: Your local changes to the following files would be overwritten by checkout: error: unable to create file specs/port/tools/gft_deep_demo.t27: Permission denied $WORKSPACE_DIR was owned by the bee, so the one-time ownership walk was skipped, while files underneath it were not - left by a root-run git from an older image. `find ! -user -print -quit` stops at the FIRST wrong file, so the healthy case costs one stat and the 45 GB walk that once outlasted the 300 s healthcheck cannot come back. The repair walks the checkout only, never the worktrees beside it, and changes only what is wrong. THE FETCH. Measured at concurrency four on one key against integrate.api.nvidia.com: two answers 200, one 503, and one socket that never answered at all. The retry wrapper handled 429 and 5xx and the 200-carrying-an- error case, and could not see the fourth - there is no response to branch on, so it reached the agent loop as a terminal error and ended the turn. A throw is now retried on the same backoff. An abort the CALLER asked for is re-thrown at once: retrying a cancelled request outlives the thing that cancelled it. bun test apps/server/src/lib/overload-retry-fetch.test.ts 10 pass, 0 fail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
❌ Tests failed — 12/2431 failed
Failed tests
|
Every tick for six minutes chose an issue and then refused to start it:
Queen tick chose an issue but the container cannot carry another bee
issue=4438 resource="disk"
Twenty lanes open, 684 candidates waiting, and zero bees - because the volume
had filled with bee worktrees nobody could use. An earlier reading of the same
volume found 41 of them holding 45 GB and three million inodes.
At entrypoint time no bee is running - this process is what starts the server
that starts them - so every directory under .worktrees/ belongs to a container
that is already gone. `git worktree prune` alone does not do it: it drops the
admin records for directories already removed, and these are still there.
Free space is printed before and after, so the next reader sees what it bought.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Measured on the running deployment after this branch was deployed, and left here because this repository has issues disabled. RING-00 answers The same answer at 17, 18 and 19 against a capacity of 20, so RING-00 is not computing a different limit — it is computing from a limit it does not have. Swift is authoritative, the swarm continues, and the disagreement is only a log line. Since the lane count is derived in Also measured in the same window, and the reason the swarm still moves: 76 |
Two failures with one symptom: an empty branch and a turn that reads as a
model failure. 80 of 101 stuck issues on 2026-09-20 had that shape.
The checkout
From the running deployment's own log:
$WORKSPACE_DIRwas owned by the bee, so the one-time ownership walk wasskipped — while files underneath it were not, left by a root-run git from an
older image. A bee that cannot write the checkout produces nothing.
find ! -user -print -quitstops at the first wrong file, so the healthycase costs one stat and the 45 GB walk that once outlasted the 300 s
healthcheck (2026-09-03) cannot come back. The repair walks the checkout only —
never the worktrees beside it — and changes only what is wrong.
The fetch
Measured at concurrency four on one key against
integrate.api.nvidia.com:The wrapper already retried 429, 5xx and the 200-carrying-an-error case. It
could not see the third one. A throw is now retried on the same backoff; an
abort the caller asked for is re-thrown at once, because retrying a
cancelled request outlives the thing that cancelled it.
Together these are what make two lanes per key viable again: the provider sheds
load about half the time at concurrency two, and every one of those sheds is now
retried rather than ending a turn.
🤖 Generated with Claude Code