[davanstrien]

Four separate fixes from a correctness review of the judge path, one commit each, plus six commits of review follow-ups. Each has tests that fail without it.

1. A budget stop kept ties from pages the run did not reach

_trim_to_budget dropped the judged comparisons over the budget but kept every auto-tie (identical outputs, no judge call) after them. With --no-adaptive --max-comparisons that was the rest of the sample.

Reproduction (300 pages, two models identical on 20% of them, budget 60): the pair had 60 ties in 78 rows (77%), and its two models were ranked on 78 pages against 21 for the third. After the fix: 3 ties in 21 rows, 21 pages each.

The run now stops at the last judged comparison that fits. A later run records the dropped ties. With --no-adaptive or balanced, and when no judge call fails, runs that each stop at a budget end with the comparisons of one run with no budget. A reviewer checked 48 chains on a board that never converges (both modes, 12 budgets): same rows, same order, no duplicate. The test suite checks 9 budgets on a board that converges. targeted chooses its pairs again at each start, so its chains can differ; that is not changed here.

2. A rerun of a converged board judged one more batch

--adaptive-strategy balanced checked the stopping rule only after a batch. It now checks the stored comparisons before the first batch, as targeted already did. A converged board makes no judge call, and the message says how to judge more (--no-adaptive).

The check is made only when the stored comparisons are exactly the first batches of the visiting order: every stored comparison is on a page of the batches before the first batch with work left. A batch that is partly judged (a budget stop, a stopped Job) is finished first, and the rule is checked at its end. So a chain of stopped runs looks at the rule where one uninterrupted run does. The check reads the stored comparisons, not the metadata: a checkpoint writes comparisons and no metadata row, so the latest row can describe an earlier run.

Two reviewers ran the same harness of 11,962 sequences on 25 boards (every budget from 1 to 35, mixed budgets, a kill at every stored prefix, blank pages, all-tie pages, error outputs), the second time on the final code: 0 differences in rows, order and verdicts against one uninterrupted run, 0 judge calls on the reruns after each, and no sequence that stopped earlier than the uninterrupted run. On 1,500 random stored states the code matched an independent implementation of the documented rule in every one.

While the stored comparisons are not such a prefix, the run judges before it checks, as on main, so a rerun still makes judge calls: a model added after the other models were judged on later pages (until its comparisons fill the earlier batches), a targeted board, judge calls that keep failing (each rerun repeats them), and results stored before 0.0.7 in another visiting order.

3. 16-bit greyscale pages reached the judge as a white image

convert("RGB") clips PIL modes I;16 (and I;16L, I;16B, I;16N), I and F at 255. The values are now divided by the image's largest value: the lightest pixel is white and zero stays black. The page is scaled in strips of rows; on a 5000×7000 page the scaling step takes about 0.2 s and 134 MiB of extra memory for I;16 (267 MiB for I and F); most of that is the pixel array that numpy reads from the image. A reviewer compared 2,684 images with a whole-array reference: 0 differences.

The first version stretched the darkest pixel to black. A reviewer measured that this turned a blank page mid-grey and gave show-through the contrast of ink, so the follow-up keeps zero as black. A second reviewer compared three mappings on real 16-bit TIFF and PNG pages (paper / ink as the judge receives them):

Page	This PR	Fixed /257	Min–max stretch
Normal	239 / 17	224 / 16	238 / 5
Blank, noise and vignette	239	220	167 (min 16)
Show-through 5% darker	250 / 238	234 / 223	199 / 53
12-bit data in a 16-bit file	246 / 15	14 / 2	246 / 6
Underexposed (9000 / 1500)	240 / 33	35 / 6	238 / 9

A high percentile in place of the maximum was not clearly better: it fixes one hot pixel, and it breaks an inverted page with little ink.

4. Failed-output counts used every dataset row, not the judged sample

With --max-samples below the dataset length, a model with an error output on every sampled page and text on some unsampled row was published with rating 1500, no interval and no comparisons, and adaptive mode could not stop. A model with error outputs only outside the sample was marked degraded.

judge now counts over the judged sample. The metadata records failed_outputs_scope: sample. share words its card from that field and does not recompute stored boards: results judged by an earlier version keep the "source dataset" wording.

FAILED still means only that every output of the model in the judged sample is an error message. A model that ends with no ranked page for another reason (a small budget stops before any page where it has text) is published as before.

Tests

• Full suite: 1125 passed; ruff, ty and rumdl pass.
• 72 new test cases.
• Mutants of the stopping check that the tests catch: check at any point, at a page end inside a batch, on the first page only, one batch early or late, at every batch; no minimum; > for the minimum; pages of out-of-grid rows counted; never check. One mutant is not covered on purpose: making the same check in targeted mode differs only on a targeted board with a pair never compared, which behaves as on main.
• Image fix: min–max stretch, divide by 65535, a removed mode and a range taken from one strip are caught.

Review before this PR

Four rounds, each by Codex and by an independent agent.

• Round 1 (the four fixes): the image commit lost precision for 32-bit integers and missed I;16N; min–max stretching made blank pages grey; checking the rule at every budget stop made budget chains stop early (451 to 472 rows where one run gave 480).
• Round 2: a gate that skipped the check after a budget stop read the latest metadata row, which a checkpoint does not update. Chains with a budget that ends on a batch ran to the end of the sample. The image conversion made several full-size copies. Two tests passed without the change they covered.
• Round 3: "no stored comparison on the pages of the first batch with work left" is not a batch boundary when the judge calls of an earlier batch all failed; the condition is now a finished prefix of the visiting order. The second reviewer found no failing sequence and six mutants that the tests did not catch.
• Round 4 (the final condition): no finding against the code from either reviewer. Codex went through twelve states against main and concluded that the condition cannot stop a run earlier than an uninterrupted run; where it does not apply, the run does extra work as on main. The agent re-ran the sequence harness (numbers above), ran 60 failed-batch cases (the run judges on the first resume in 60 of 60, where the round-3 code made no call in 13 of 15), and 56 clean converged states (0 calls on every rerun). Its three nits, two missing tests and one sentence, are in the last commit.

Known, not changed here

• audit counts error outputs over the whole dataset; judge counts over the sample.
• A rerun that judges nothing still appends a metadata row, and share takes its "Judged" date from the last row.
• A float image with values from 0 to 1 is still sent as black. A dark page with one bright outlier (a hot pixel, a border, glare) stays dark, and a 32-bit page with a no-data marker far below zero becomes a white page; main sends white for all of these.
• A run that uses up its budget exactly when the rule is met is recorded with budget_exhausted: true.
• The image conversion is not part of the resume provenance.
• The test suite makes two read-only Hub requests (a repo_exists call from the judge test harness and one public file download).

🤖 Generated with Claude Code