Skip to content

fix(queen): the swarm read GitHub anonymously and died on 403 half the day - #476

Open
gHashTag wants to merge 4 commits into
feat/queen-supervisorfrom
fix/queen-round-latency
Open

gHashTag wants to merge 4 commits into
feat/queen-supervisorfrom
fix/queen-round-latency

Conversation

@gHashTag

@gHashTag gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner

The bees stop because the round dies. This is why, measured.

Measured, over the last twelve hours of production

minutes with zero bees working 357 of 721 (50%)
minutes with all four working 258 (36%)
median gap between bursts of work 22 min, against a 5-minute tick
longest gap 41.7 min
rounds ending GitHub returned 403 in one log window 144

The shape is bimodal — either four bees or none:

0 bees: 357 min (50%)
1 bees:  42 min  (6%)
2 bees:  19 min  (3%)
3 bees:  45 min  (6%)
4 bees: 258 min (36%)

A round either gets its issue list or dies whole. openIssues throws on a bad status, so one 403 takes the entire round with it — no review, no choice, no dispatch — and the capacity sits idle until some later round happens to get through.

The log makes it plain:

05:47:16 Queen tick decided → dispatch
05:50:32 Queen round starting   (no decision)
05:51:11 Queen round starting → Queen tick round failed  {"error":"GitHub returned 403"}
05:54:21 Queen round starting → Queen tick round failed  {"error":"GitHub returned 403"}
05:56:12 Queen round starting → Queen tick round failed  {"error":"GitHub returned 403"}
06:01:12 Queen round starting → Queen tick round failed  {"error":"GitHub returned 403"}
06:06:12 Queen round starting → Queen tick round failed  {"error":"GitHub returned 403"}
06:11:11 Queen round starting → decided, dispatched 4

The dispatch loop is not the problem — when a round survives, it fills all four slots in five seconds.

The cause

Every GitHub read in this file went out unauthenticated. The anonymous limit is 60 requests an hour, and this file has said so in a comment since it was written:

"a second round trip per candidate against an anonymous rate limit that is 60 an hour"

The limit was designed around instead of lifted. openIssues pages the backlog and bodiesFor fetches one body per candidate — twelve rounds an hour against a budget of sixty.

The token was in the environment the whole time. GH_TOKEN is set on this service and /rate_limit answers 15000 of 15000 remaining — the measurement that turns "we are being throttled" into "we are throttled at the anonymous tier while holding a key to the other one."

60/h → 15000/h.

What this is not

It hands nothing to a bee. This is the supervisor's own outbound read. The worker environment is still built from the ten-entry allowlist that deliberately excludes the GitHub token — untouched.

Falls back to anonymous when no token is set, so a local run without secrets behaves as it always has.

Tests

They pin the header, not the fetch — a test that mocked fetch would pass against a call that still forgot to ask for these headers. GH_TOKEN is checked first by name, because reading only GITHUB_TOKEN would leave the limit at sixty an hour on the one deployment that matters while every test still passed.

Verified locally by extracting the real helper from the committed source (this checkout cannot run the full server suite): 3 pass, 0 fail. CI is the witness for the suite.

🤖 Generated with Claude Code

…e day

MEASURED, over the last twelve hours of production:

  minutes with ZERO bees working : 357 of 721  (50%)
  minutes with all four working  : 258         (36%)
  median gap between bursts      : 22 min against a 5-minute tick
  longest gap                    : 41.7 min
  rounds ending "GitHub returned 403" in one log window : 144

The shape is bimodal - either four bees or none - because a round either gets
its issue list or dies whole. `openIssues` throws on a bad status, so a single
403 takes the entire round with it: no review, no choice, no dispatch, and the
capacity sits idle until some later round happens to get through.

THE CAUSE. Every GitHub read in this file went out unauthenticated. The
anonymous limit is sixty requests an hour, and this file has said so in a
comment since it was written - "a second round trip per candidate against an
anonymous rate limit that is 60 an hour". The limit was designed around instead
of lifted: `openIssues` pages the backlog and `bodiesFor` fetches one body per
candidate, twelve rounds an hour, against a budget of sixty.

The token was in the environment the whole time. `GH_TOKEN` is set on this
service, and `/rate_limit` answers 15000 of 15000 remaining - the measurement
that turns "we are being throttled" into "we are throttled at the anonymous
tier while holding a key to the other one". 60/h becomes 15000/h.

WHAT THIS IS NOT. It hands nothing to a bee. This is the supervisor's own
outbound read. The worker environment is still built from the ten-entry
allowlist that deliberately excludes the GitHub token, and that is untouched.

Falls back to anonymous when no token is set, so a local run without secrets
behaves as it always has rather than failing to start.

The tests pin the HEADER, not the fetch: a test that mocked `fetch` would pass
against a call that still forgot to ask for these headers. `GH_TOKEN` is
checked first by name, because reading only `GITHUB_TOKEN` would leave the
limit at sixty an hour on the one deployment that matters while every test
still passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

✅ Tests passed — 2064/2096

Suite Passed Failed Skipped
agent 87/87 0 0
build 9/9 0 0
cdp-protocol 5/5 0 0
eval 93/93 0 0
server-agent 272/272 0 0
server-api 951/982 0 31
server-browser 6/6 0 0
server-integration 10/11 0 1
server-lib 273/273 0 0
server-pglive 3/3 0 0
server-root 68/68 0 0
server-skills 31/31 0 0
server-tools 242/242 0 0
shared 14/14 0 0

View workflow run

Measured from inside the container while writing this: the service burns 77
anonymous GitHub requests an hour against a limit of 60. The first commit said
"twelve rounds an hour against a budget of sixty" and understated it - the
round makes about six calls, not two. Once the budget is spent every call
returns 403 until the hourly reset, which is why the longest observed idle
stretch was 41.7 minutes: it is the reset window, not a coincidence.

  anonymous : 200  remaining 23/60,  resets in 2352s
  with token: 200  remaining 14813/15000

TWO CORRECTIONS TO MY OWN FIRST COMMIT.

The premise was not neglect, it was expiry, and the difference matters because
the reasoning was good. The doc comment said: "Anonymous on purpose: the
repository is public, this is a read, and a token here would be a credential in
a container for no gain. GitHub's anonymous rate limit is 60/hour against a
loop that ticks at most A FEW TIMES AN HOUR." The loop ticks twelve times an
hour now. The sentence outlived its assumption, and it is kept in place, quoted,
rather than deleted - a rule that expired teaches more than one that was wrong.
The stated cost was also already paid: `GH_TOKEN` is on the SUPERVISOR, and the
worker allowlist that excludes it is untouched.

And the first commit's own patch was sloppy: inserting the helper immediately
above `openIssues` split that function from its doc comment, leaving the
paragraph about pagination and `complete` documenting a headers helper. Moved.

THE SECOND DEFECT, which the token alone does not fix. `openIssues` threw on any
bad status, so ONE refusal took the entire round: no review, no choice, no
dispatch. 135 of 136 round failures were exactly that. But this function already
has a word for a partial answer - `complete: false`, which `rememberIssues` is
built to respect, precisely so a truncated list is "still worth deciding
against but never treated as the whole truth". A refusal on page two is that
case. It now breaks and reports the pages it did get.

A refusal on page ONE still throws: a round with no issue list has nothing to
decide against, and tolerating it would dispatch against an empty board.

The tests replace `globalThis.fetch` and restore it rather than using
`mock.module`, which is process-global in bun and cannot be undone - a fake left
behind here would be inherited by every test file that ran after it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag

gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner Author

Two corrections to this PR, and a second defect the token alone does not fix

(Reposted: the first version of this comment lost every backticked identifier to shell command substitution. My mistake, and a good argument for --body-file.)

The burn rate, measured rather than estimated

The description said "twelve rounds an hour against a budget of sixty". Measured from inside the container while writing the follow-up:

t0  anonymous remaining 23/60, resets in 2352s
t1  anonymous remaining 14/60   (7 min later)
    burned 9 requests in 7 min  ->  about 77/hour

77 an hour against a limit of 60. The round makes about six calls, not two. And the paging loop does break on a short page, so my arithmetic was wrong in both directions — the real number is worse.

Once the budget is spent, every call 403s until the hourly reset. That is why the longest observed idle stretch was 41.7 minutes — it is the reset window, not a coincidence.

Side by side, right now:

anonymous : 200  remaining 23/60
with token: 200  remaining 14813/15000

It was expiry, not neglect

I framed this as a limit the author designed around instead of lifting. The doc comment is more honest than my framing:

"Anonymous on purpose: the repository is public, this is a read, and a token here would be a credential in a container for no gain. GitHub's anonymous rate limit is 60/hour against a loop that ticks at most a few times an hour."

The reasoning was sound. The loop ticks twelve times an hour now. The sentence outlived its assumption. It is kept in place and quoted rather than deleted — a rule that expired teaches more than one that was wrong.

The stated cost was also already paid: GH_TOKEN is on the supervisor, not a bee, and the worker allowlist that excludes it is untouched.

My own patch was sloppy

Inserting the helper immediately above openIssues split that function from its doc comment, leaving the paragraph about pagination and complete documenting a headers helper. Moved.

The second defect: one 403 killed the whole round

The token fixes the rate limit. It does not fix the fragility. openIssues threw on any bad status, so one refusal took the entire round — no review, no choice, no dispatch. 135 of 136 round failures were exactly that.

But this function already has a word for a partial answer: complete: false, which rememberIssues is built to respect, precisely so a truncated list is "still worth deciding against but never treated as the whole truth". A refusal on page two is that case. It now breaks and reports the pages it did get.

A refusal on page one still throws — a round with no issue list has nothing to decide against, and tolerating it would dispatch against an empty board.

Tests replace globalThis.fetch and restore it rather than using mock.module, which is process-global in bun and cannot be undone.

The same shape as the 403, one layer in. Review and reaping are housekeeping;
dispatch is the thing the hive exists to do. An exception in either took the
whole round with it, so a transient database error cost five minutes of every
bee - and one round measured on 2026-09-06 died exactly that way, on
`deadlock detected`, which nothing in this repository had ever mentioned.

An audit of the round body says these were the last two: every other awaited
call on the path is already guarded with `.catch(() => null)`. `openIssues` was
the third and is fixed in the commit before this one.

NEITHER IS LOST BY CONTINUING. The review re-reads every unjudged dispatch next
round by construction - that is what `review_state IS NULL OR = 'wait'` means -
and the reaper re-finds a stalled dispatch. What IS lost by throwing is the
dispatch that would have happened, and that is the one thing a later round
cannot give back: the idle minutes are already spent.

Logged at warn with the reason rather than swallowed. `tri idle` reads these
lines out of the service log and reports what stopped the rounds, so a review
that fails EVERY round is loud rather than merely survivable.

WHAT IS NOT TESTED HERE, said plainly: the catch itself has no unit test.
`runQueenTickOnce` needs a live database to drive, and the round test file
exercises the pieces rather than the whole. CI's type check covers the fallback
shapes - `ReviewRound` for one, `Awaited<ReturnType<...>>` for the other - and
the failure this fixes is evidenced in the production log rather than in a
fixture. A live-Postgres round test belongs in the `server-pglive` job and is
not in this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `.catch()` in the commit before this one had no test, and I said so - along
with a reason that was wrong. I claimed driving `runRound` needed a live
database. It does not: `runRound` is exported, takes a pool double, and
`queen-round.test.ts` has had a recording fake, a stubbed `fetch` and the real
policy binary since it was written. I asserted the file's limits without
reading it.

So here is the test. `FROM queen_dispatch d` is the review's SELECT and the
only query in the round using that alias, so failing it targets the review and
nothing else. The round is then expected to reach `dispatchBee` anyway and
record its INSERT. Reverting the `.catch()` turns this red: the round stops
before the dispatch and the INSERT never appears.

The shared fake gained a `throwOn` parameter rather than the test monkey-
patching `pool.query`, which needed a cast through `unknown` to compile and
would have left the next reader wondering which of the two queries recorded.
The failing statement is recorded BEFORE it throws, because a query that throws
is still a query the round issued.

HONEST LIMIT, and it is why this is a separate commit: these cases are guarded
by `it.if(present)` and the policy binary is not built in CI, so this test
SKIPS there today along with the other seven. #477 builds it and makes the skip
loud. Until that lands, this is proven on a machine with `queen-core` built and
nowhere else - which is exactly the state #477 exists to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant