Skip to content

fix(queen): the Queen's own slot numbers made her unable to see the answers - #472

Open
gHashTag wants to merge 2 commits into
feat/queen-supervisorfrom
loop/verdict-number-prefix
Open

gHashTag wants to merge 2 commits into
feat/queen-supervisorfrom
loop/verdict-number-prefix

Conversation

@gHashTag

@gHashTag gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner

The finding

153 dispatches sit in sendBack having spent no retry budget. Not one has a single criterion that was tested and failed. Every one carries a complete, parseable ## VERDICT block. They were returned for criteria they had already answered.

The cause

The brief hands the bee numbered slots and asks for - 1. <criterion>: met, so parseVerdictBlock returns a criterion beginning "1. ". The promised text carries no number.

unjudgedCriteria documents matching by containment in either direction:

a criterion counts as judged when one side contains the other — containment in EITHER direction, because the 300-character slice means the promised text may contain the line's text and not the other way round

One of those two directions could not fire even once. want.includes(line) is impossible when the line always carries a prefix the promise lacks. Only line.includes(want) survived, and that demands the bee reproduce the criterion whole. Shorten it by one trailing sentence and the criterion reads as never answered.

Because "unanswered" is deliberately not charged against the retry ceiling (browseros-ai#1420 FR-003), these never escalate and never stop. That is the 80-issue ceiling-less loop reported in #471, and this is its cause.

The measurement

Re-judged all 153 with the review's own functions, against the live board:

no verdict lines at all : 0
every criterion judged  : 0
some unjudged           : 153
   76  promised=5 verdictLines=5 unjudged=1 failed=0
   31  promised=4 verdictLines=4 unjudged=1 failed=0
   27  promised=5 verdictLines=5 unjudged=2 failed=0
   ...
unjudged>0 with the CURRENT matcher        : 153
unjudged>0 after stripping the "N. " prefix:   5
fixed by stripping alone                   : 148

148 of 153. The remaining five are unjudged for reasons of their own.

Real example (browseros-ai#1527) — promise vs. what the bee wrote:

promised: `LC_ALL=C grep -cP ...` prints 0, and the raw output is quoted in the
          report. The command MUST NOT name or enumerate the specific items ...
wrote   : 1. `LC_ALL=C grep -cP ...` prints 0, and the raw output is quoted in
          the report

The repository already had the right answer

missingVerdictSlots strips the number and matches by slot. Run over these same 153 it reports every one complete. Its own comment claims it and the review "read the SAME block ... so a line this counts as answered is exactly a line the review counts as judged." That was false — two implementations of one rule, disagreeing, with a comment asserting they could not.

Why the suite missed it

There was already a case for each direction of the match, including the truncated-quote case. Every one used an unnumbered criterion, and production has not sent an unnumbered one since the brief began numbering slots. A numbered line that quotes its criterion in full still matches on the old code, so only the shortening shape fails: the numbered cases were never written, and the shortening cases were never numbered.

The change

One line, inside normalize, applied to both sides so this stays one rule rather than a special case at a call site. The filter returns the original strings, so nothing a person reads loses its number.

Verification

Extracted the real unjudgedCriteria from the committed blob and ran it directly (the function is self-contained, so it needs none of the server's deps — this checkout cannot run the full suite):

  • before the fix: 2 fail, 5 pass
  • after the fix: 7 pass, 0 fail

The two failing cases are the discriminating ones. The other five are labelled guards and pass on both sides, so the strip cannot be bought at the cost of the containment fence — in particular a short numbered line still does not judge a long criterion, and a criterion opening with a decimal keeps its meaning.

CI is the witness for the full suite.

🤖 Generated with Claude Code

…nswers

153 dispatches sit in sendBack having spent no retry budget, and not one of
them has a single criterion that was tested and FAILED. Every one carries a
complete, parseable VERDICT block. They were returned for criteria they had
answered.

The brief hands the bee numbered slots and asks for `- 1. <criterion>: met`,
so `parseVerdictBlock` returns a criterion beginning "1. ". The promised text
carries no such number. `unjudgedCriteria` documents matching by containment
in EITHER direction - and one of those two directions could not fire even
once, because the line always held a prefix the promise did not. Only
`line.includes(want)` survived, and that demands the bee reproduce the
criterion WHOLE. Shorten it by one trailing sentence and the criterion reads
as never answered.

Measured over all 153: stripping the prefix clears 148. The remaining five are
unjudged for reasons of their own.

Stripped on BOTH sides, inside `normalize`, so this stays one rule instead of a
special case at a call site. The filter returns the ORIGINAL strings, so
nothing a person reads loses its number.

The repository already had the right answer and did not know it.
`missingVerdictSlots` strips the number and matches by slot; run over these
same 153 it reports every one COMPLETE. Its comment claims it and the review
"read the SAME block ... so a line this counts as answered is exactly a line
the review counts as judged". That was false: two implementations of one rule,
disagreeing, with a comment asserting they could not.

WHY THE SUITE MISSED IT. There was already a case for each direction of the
match, including the truncated-quote case. Every one used an UNNUMBERED
criterion, and production has not sent an unnumbered one since the brief began
numbering slots. A numbered line that quotes its criterion in full still
matches on the old code, so only the shortening shape fails - the numbered
cases were never written and the shortening cases were never numbered.

Two new cases fail before this change and pass after; five more are labelled
guards and pass on both sides, so the strip cannot be bought at the cost of
the containment fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the fix label Sep 6, 2026
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

✅ Tests passed — 2064/2095

Suite Passed Failed Skipped
agent 87/87 0 0
build 9/9 0 0
cdp-protocol 5/5 0 0
eval 93/93 0 0
server-agent 272/272 0 0
server-api 951/981 0 30
server-browser 6/6 0 0
server-integration 10/11 0 1
server-lib 273/273 0 0
server-pglive 3/3 0 0
server-root 68/68 0 0
server-skills 31/31 0 0
server-tools 242/242 0 0
shared 14/14 0 0

View workflow run

…the board

I wrote that "the repository already had the right answer and did not know it",
meaning `missingVerdictSlots`. Measured across the whole board, that is wrong
and worth correcting before someone acts on it.

Of 358 verdict blocks with criteria, 156 carry NO numbering at all, and slot
matching finds NOTHING covered on every one of those 156. Wiring it as the
review's matcher would mark every criterion in 44% of the board unanswered -
the defect this branch fixes, with a wider blast radius.

The two functions disagree on 347 of 358 rows, in BOTH directions. Neither is
right alone: text matching was blind to the number, slot matching is blind to
its absence. Stripping the number and then matching by text is the only rule
that reads both the numbered briefs and the unnumbered ones, which is why the
fix is a strip and not a swap. Said so at the call site, as a DO NOT SIMPLIFY
note, because the resemblance is what makes the wrong move attractive.

Also removed the false sentence from `missingVerdictSlots` itself: "one parser,
so a line this counts as answered is exactly a line the review counts as
judged". Sharing a parser is not sharing a rule. That sentence is what made the
numbered prefix look like somebody else's already-solved problem, and it is why
the defect kept its cover.

No behaviour change; comments only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag

gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner Author

Correcting my own claim in this PR

The description says "The repository already had the right answer and did not know it", meaning missingVerdictSlots. That is wrong, and I'd rather correct it than have someone act on it.

Measured across the whole board (358 verdict blocks with criteria):

blocks whose lines carry a N. prefix 202
blocks with no numbering at all 156
…of those, where slot matching finds nothing covered 156 of 156

So slot matching is blind on 44% of the board. Wiring it as the review's matcher would mark every criterion in those 156 unanswered — this PR's defect with a wider blast radius.

The two functions disagree on 347 of 358 rows, in both directions:

  76  sendBack: bySlot=5 byText=4 of 5    <- the defect this PR fixes
  44  accept:   bySlot=0 byText=4 of 4    <- slot matching blind on unnumbered
  34  accept:   bySlot=0 byText=5 of 5
  32  sendBack: bySlot=4 byText=3 of 4

Neither is right alone. Text matching was blind to the number; slot matching is blind to its absence. Stripping the number and then matching by text is the only rule that reads both the numbered briefs and the unnumbered ones — so the fix is a strip, not a swap.

The diff is unchanged; this adds two comments:

  • a DO NOT SIMPLIFY INTO SLOT MATCHING note at the call site, because the resemblance is exactly what makes the wrong move attractive later
  • removal of the false sentence in missingVerdictSlots: "one parser, so a line this counts as answered is exactly a line the review counts as judged". Sharing a parser is not sharing a rule, and that sentence is what made this defect look like somebody else's already-solved problem.

Found by building the agreement gate that this round set out to write — the gate's first run corrected the round that motivated it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant