Skip to content

A bee that cannot write the checkout, and a fetch that threw - #491

Merged
gHashTag merged 2 commits into
feat/queen-supervisorfrom
fix/checkout-ownership
Sep 21, 2026
Merged

gHashTag merged 2 commits into
feat/queen-supervisorfrom
fix/checkout-ownership

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Two failures with one symptom: an empty branch and a turn that reads as a
model failure.
80 of 101 stuck issues on 2026-09-20 had that shape.

The checkout

From the running deployment's own log:

error: Your local changes to the following files would be overwritten by checkout:
error: unable to create file specs/port/tools/gft_deep_demo.t27: Permission denied

$WORKSPACE_DIR was owned by the bee, so the one-time ownership walk was
skipped — while files underneath it were not, left by a root-run git from an
older image. A bee that cannot write the checkout produces nothing.

find ! -user -print -quit stops at the first wrong file, so the healthy
case costs one stat and the 45 GB walk that once outlasted the 300 s
healthcheck (2026-09-03) cannot come back. The repair walks the checkout only —
never the worktrees beside it — and changes only what is wrong.

The fetch

Measured at concurrency four on one key against integrate.api.nvidia.com:

request 0: 200  3.2s
request 1: 503  0.9s  {"error":{"message":"Service temporarily overloaded"}}
request 2: ERR 60.3s  read timed out          <-- no response to branch on
request 3: 200  5.3s

The wrapper already retried 429, 5xx and the 200-carrying-an-error case. It
could not see the third one. A throw is now retried on the same backoff; an
abort the caller asked for is re-thrown at once, because retrying a
cancelled request outlives the thing that cancelled it.

$ bun test apps/server/src/lib/overload-retry-fetch.test.ts
10 pass, 0 fail

Together these are what make two lanes per key viable again: the provider sheds
load about half the time at concurrency two, and every one of those sheds is now
retried rather than ending a turn.

🤖 Generated with Claude Code

TWO FAILURES WITH ONE SYMPTOM: an empty branch and a turn that reads as a model
failure. 80 of 101 stuck issues on 2026-09-20 had exactly that shape.

THE CHECKOUT. Measured in the running deployment's log:

  error: Your local changes to the following files would be overwritten by checkout:
  error: unable to create file specs/port/tools/gft_deep_demo.t27: Permission denied

$WORKSPACE_DIR was owned by the bee, so the one-time ownership walk was skipped,
while files underneath it were not - left by a root-run git from an older image.
`find ! -user -print -quit` stops at the FIRST wrong file, so the healthy case
costs one stat and the 45 GB walk that once outlasted the 300 s healthcheck
cannot come back. The repair walks the checkout only, never the worktrees
beside it, and changes only what is wrong.

THE FETCH. Measured at concurrency four on one key against
integrate.api.nvidia.com: two answers 200, one 503, and one socket that never
answered at all. The retry wrapper handled 429 and 5xx and the 200-carrying-an-
error case, and could not see the fourth - there is no response to branch on,
so it reached the agent loop as a terminal error and ended the turn. A throw is
now retried on the same backoff. An abort the CALLER asked for is re-thrown at
once: retrying a cancelled request outlives the thing that cancelled it.

  bun test apps/server/src/lib/overload-retry-fetch.test.ts
  10 pass, 0 fail

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown

❌ Tests failed — 12/2431 failed

Suite Passed Failed Skipped
agent 87/87 0 0
build 9/9 0 0
cdp-protocol 5/5 0 0
eval 93/93 0 0
server-agent 272/272 0 0
server-api 1247/1311 7 57
server-browser 6/6 0 0
server-integration 10/11 0 1
server-lib 279/279 0 0
server-pglive 1/3 2 0
server-root 68/68 0 0
server-skills 31/31 0 0
server-tools 239/242 3 0
shared 14/14 0 0
Failed tests
  • server-apithe salvage commit > never splits a rename across the path cap
  • server-apithe note says who committed the branch, where it can be read > puts the salvage fact where the 1500-character cap cannot cut it
  • server-apia turn the provider killed does not spend the issue > is judged and sent back, but charges neither counter
  • server-apideriving work the repository already measured > carries the command that produced it
  • server-apideriving work the repository already measured > gives every candidate a path that can be a boundary
  • server-apideriving work the repository already measured > proposes nothing from code this project does not own
  • server-apisalvaging a turn that ended with its work uncommitted > refuses a worktree holding an unmerged path
  • server-pglivethe migration block, applied to a real PostgreSQL > creates every object it promises, and survives a second boot
  • server-pglivethe live gate > ran, or its absence is on the record
  • server-toolsnavigation tools > new_hidden_page opens a hidden tab
  • server-toolsnavigation tools > show_page restores a hidden page to visible
  • server-toolswindow tools > create_hidden_window creates and closes a hidden window

View workflow run

Every tick for six minutes chose an issue and then refused to start it:

  Queen tick chose an issue but the container cannot carry another bee
    issue=4438 resource="disk"

Twenty lanes open, 684 candidates waiting, and zero bees - because the volume
had filled with bee worktrees nobody could use. An earlier reading of the same
volume found 41 of them holding 45 GB and three million inodes.

At entrypoint time no bee is running - this process is what starts the server
that starts them - so every directory under .worktrees/ belongs to a container
that is already gone. `git worktree prune` alone does not do it: it drops the
admin records for directories already removed, and these are still there.

Free space is printed before and after, so the next reader sees what it bought.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag

Copy link
Copy Markdown
Owner Author

Measured on the running deployment after this branch was deployed, and left here because this repository has issues disabled.

RING-00 answers freeSlots=0 whatever the running count. Six lines in one minute, 2026-09-20T14:54Z:

RING-00 DISAGREES WITH THE SWIFT POLICY
  question="canStartAnother" running=17 swift=true ring=false freeSlots="0"
  ... running=18 swift=true ring=false freeSlots="0"
  ... running=19 swift=true ring=false freeSlots="0"

The same answer at 17, 18 and 19 against a capacity of 20, so RING-00 is not computing a different limit — it is computing from a limit it does not have. Swift is authoritative, the swarm continues, and the disagreement is only a log line.

Since the lane count is derived in docker-entrypoint.sh and exported into the environment both readers share, the likely gap is that t27core is baked into the image and reads its own default instead. Worth a separate change: the cross-check is the only thing standing between the two implementations and the failure this repository has already had (capacity 12 reported while every tick refused with "limit 4"), and a disagreement that is always the same and always ignored has stopped being evidence.

Also measured in the same window, and the reason the swarm still moves: 76 OpenAI-compatible endpoint overloaded, retrying lines in 400 log lines. The retry added here is absorbing a provider that sheds load constantly at twenty concurrent bees on ten keys.

@gHashTag
gHashTag merged commit 7f49a8b into feat/queen-supervisor Sep 21, 2026
12 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants