Conversation
…e day MEASURED, over the last twelve hours of production: minutes with ZERO bees working : 357 of 721 (50%) minutes with all four working : 258 (36%) median gap between bursts : 22 min against a 5-minute tick longest gap : 41.7 min rounds ending "GitHub returned 403" in one log window : 144 The shape is bimodal - either four bees or none - because a round either gets its issue list or dies whole. `openIssues` throws on a bad status, so a single 403 takes the entire round with it: no review, no choice, no dispatch, and the capacity sits idle until some later round happens to get through. THE CAUSE. Every GitHub read in this file went out unauthenticated. The anonymous limit is sixty requests an hour, and this file has said so in a comment since it was written - "a second round trip per candidate against an anonymous rate limit that is 60 an hour". The limit was designed around instead of lifted: `openIssues` pages the backlog and `bodiesFor` fetches one body per candidate, twelve rounds an hour, against a budget of sixty. The token was in the environment the whole time. `GH_TOKEN` is set on this service, and `/rate_limit` answers 15000 of 15000 remaining - the measurement that turns "we are being throttled" into "we are throttled at the anonymous tier while holding a key to the other one". 60/h becomes 15000/h. WHAT THIS IS NOT. It hands nothing to a bee. This is the supervisor's own outbound read. The worker environment is still built from the ten-entry allowlist that deliberately excludes the GitHub token, and that is untouched. Falls back to anonymous when no token is set, so a local run without secrets behaves as it always has rather than failing to start. The tests pin the HEADER, not the fetch: a test that mocked `fetch` would pass against a call that still forgot to ask for these headers. `GH_TOKEN` is checked first by name, because reading only `GITHUB_TOKEN` would leave the limit at sixty an hour on the one deployment that matters while every test still passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Tests passed — 2064/2096
|
Measured from inside the container while writing this: the service burns 77 anonymous GitHub requests an hour against a limit of 60. The first commit said "twelve rounds an hour against a budget of sixty" and understated it - the round makes about six calls, not two. Once the budget is spent every call returns 403 until the hourly reset, which is why the longest observed idle stretch was 41.7 minutes: it is the reset window, not a coincidence. anonymous : 200 remaining 23/60, resets in 2352s with token: 200 remaining 14813/15000 TWO CORRECTIONS TO MY OWN FIRST COMMIT. The premise was not neglect, it was expiry, and the difference matters because the reasoning was good. The doc comment said: "Anonymous on purpose: the repository is public, this is a read, and a token here would be a credential in a container for no gain. GitHub's anonymous rate limit is 60/hour against a loop that ticks at most A FEW TIMES AN HOUR." The loop ticks twelve times an hour now. The sentence outlived its assumption, and it is kept in place, quoted, rather than deleted - a rule that expired teaches more than one that was wrong. The stated cost was also already paid: `GH_TOKEN` is on the SUPERVISOR, and the worker allowlist that excludes it is untouched. And the first commit's own patch was sloppy: inserting the helper immediately above `openIssues` split that function from its doc comment, leaving the paragraph about pagination and `complete` documenting a headers helper. Moved. THE SECOND DEFECT, which the token alone does not fix. `openIssues` threw on any bad status, so ONE refusal took the entire round: no review, no choice, no dispatch. 135 of 136 round failures were exactly that. But this function already has a word for a partial answer - `complete: false`, which `rememberIssues` is built to respect, precisely so a truncated list is "still worth deciding against but never treated as the whole truth". A refusal on page two is that case. It now breaks and reports the pages it did get. A refusal on page ONE still throws: a round with no issue list has nothing to decide against, and tolerating it would dispatch against an empty board. The tests replace `globalThis.fetch` and restore it rather than using `mock.module`, which is process-global in bun and cannot be undone - a fake left behind here would be inherited by every test file that ran after it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two corrections to this PR, and a second defect the token alone does not fix(Reposted: the first version of this comment lost every backticked identifier to shell command substitution. My mistake, and a good argument for The burn rate, measured rather than estimatedThe description said "twelve rounds an hour against a budget of sixty". Measured from inside the container while writing the follow-up: 77 an hour against a limit of 60. The round makes about six calls, not two. And the paging loop does break on a short page, so my arithmetic was wrong in both directions — the real number is worse. Once the budget is spent, every call 403s until the hourly reset. That is why the longest observed idle stretch was 41.7 minutes — it is the reset window, not a coincidence. Side by side, right now: It was expiry, not neglectI framed this as a limit the author designed around instead of lifting. The doc comment is more honest than my framing:
The reasoning was sound. The loop ticks twelve times an hour now. The sentence outlived its assumption. It is kept in place and quoted rather than deleted — a rule that expired teaches more than one that was wrong. The stated cost was also already paid: My own patch was sloppyInserting the helper immediately above The second defect: one 403 killed the whole roundThe token fixes the rate limit. It does not fix the fragility. But this function already has a word for a partial answer: A refusal on page one still throws — a round with no issue list has nothing to decide against, and tolerating it would dispatch against an empty board. Tests replace |
The same shape as the 403, one layer in. Review and reaping are housekeeping; dispatch is the thing the hive exists to do. An exception in either took the whole round with it, so a transient database error cost five minutes of every bee - and one round measured on 2026-09-06 died exactly that way, on `deadlock detected`, which nothing in this repository had ever mentioned. An audit of the round body says these were the last two: every other awaited call on the path is already guarded with `.catch(() => null)`. `openIssues` was the third and is fixed in the commit before this one. NEITHER IS LOST BY CONTINUING. The review re-reads every unjudged dispatch next round by construction - that is what `review_state IS NULL OR = 'wait'` means - and the reaper re-finds a stalled dispatch. What IS lost by throwing is the dispatch that would have happened, and that is the one thing a later round cannot give back: the idle minutes are already spent. Logged at warn with the reason rather than swallowed. `tri idle` reads these lines out of the service log and reports what stopped the rounds, so a review that fails EVERY round is loud rather than merely survivable. WHAT IS NOT TESTED HERE, said plainly: the catch itself has no unit test. `runQueenTickOnce` needs a live database to drive, and the round test file exercises the pieces rather than the whole. CI's type check covers the fallback shapes - `ReviewRound` for one, `Awaited<ReturnType<...>>` for the other - and the failure this fixes is evidenced in the production log rather than in a fixture. A live-Postgres round test belongs in the `server-pglive` job and is not in this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `.catch()` in the commit before this one had no test, and I said so - along with a reason that was wrong. I claimed driving `runRound` needed a live database. It does not: `runRound` is exported, takes a pool double, and `queen-round.test.ts` has had a recording fake, a stubbed `fetch` and the real policy binary since it was written. I asserted the file's limits without reading it. So here is the test. `FROM queen_dispatch d` is the review's SELECT and the only query in the round using that alias, so failing it targets the review and nothing else. The round is then expected to reach `dispatchBee` anyway and record its INSERT. Reverting the `.catch()` turns this red: the round stops before the dispatch and the INSERT never appears. The shared fake gained a `throwOn` parameter rather than the test monkey- patching `pool.query`, which needed a cast through `unknown` to compile and would have left the next reader wondering which of the two queries recorded. The failing statement is recorded BEFORE it throws, because a query that throws is still a query the round issued. HONEST LIMIT, and it is why this is a separate commit: these cases are guarded by `it.if(present)` and the policy binary is not built in CI, so this test SKIPS there today along with the other seven. #477 builds it and makes the skip loud. Until that lands, this is proven on a machine with `queen-core` built and nowhere else - which is exactly the state #477 exists to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured, over the last twelve hours of production
GitHub returned 403in one log windowThe shape is bimodal — either four bees or none:
A round either gets its issue list or dies whole.
openIssuesthrows on a bad status, so one 403 takes the entire round with it — no review, no choice, no dispatch — and the capacity sits idle until some later round happens to get through.The log makes it plain:
The dispatch loop is not the problem — when a round survives, it fills all four slots in five seconds.
The cause
Every GitHub read in this file went out unauthenticated. The anonymous limit is 60 requests an hour, and this file has said so in a comment since it was written:
The limit was designed around instead of lifted.
openIssuespages the backlog andbodiesForfetches one body per candidate — twelve rounds an hour against a budget of sixty.The token was in the environment the whole time.
GH_TOKENis set on this service and/rate_limitanswers 15000 of 15000 remaining — the measurement that turns "we are being throttled" into "we are throttled at the anonymous tier while holding a key to the other one."60/h → 15000/h.What this is not
It hands nothing to a bee. This is the supervisor's own outbound read. The worker environment is still built from the ten-entry allowlist that deliberately excludes the GitHub token — untouched.
Falls back to anonymous when no token is set, so a local run without secrets behaves as it always has.
Tests
They pin the header, not the fetch — a test that mocked
fetchwould pass against a call that still forgot to ask for these headers.GH_TOKENis checked first by name, because reading onlyGITHUB_TOKENwould leave the limit at sixty an hour on the one deployment that matters while every test still passed.Verified locally by extracting the real helper from the committed source (this checkout cannot run the full server suite): 3 pass, 0 fail. CI is the witness for the suite.
🤖 Generated with Claude Code