[davanstrien]

Why

The board card told readers that two models with overlapping 95% intervals are not clearly separated. On boards of three or more models that test is stricter than its 95% label: in simulations it called equally good models apart about 1% of the time. Testing the difference between two ratings directly, on the same resampled pages, separated 4 to 16 points more of the pairs that do differ. On a two-model board the two tests gave the same answer.

The card also did not say how many pages each pair rests on, or how often the judge read a cut output.

What this changes

Rank range (elo.py, share.py)

• compute_elo keeps the bootstrap replicates long enough to test each pair's rating difference on the same resamples. Each ranked model gets a rank range: first rank = 1 + the number of models that test clearly above it, last rank = n − the number that test clearly below. Leaderboard.rank_range holds it.
• "Clearly above" means the 95% percentile interval of the paired difference is above zero.
• A model on fewer than 20 pages is not tested against any model, which can only widen the ranges. A board with fewer than two models on 20 pages has no rank ranges, and there are none from fewer than 100 resamples.
• The range is widened, if needed, to include the model's rank in the table.
• The published intervals do not change: the same resamples produce them, with the same index arithmetic.
• The adaptive stopping rule and the 10-page minimum for a slice board are not changed.

Card (share.py)

• A "Rank range" column, and a paragraph that says how to read it and what it does not say. Each sentence is a definition, something that always follows from it (a range of one rank means the model tested clearly apart from every other ranked model; an untested model can only widen a range), or a measurement named as one ("in simulations, 4 to 7 in 100 pairs of equally good models tested apart by chance").
• "Pages per pair": the smallest and largest number of pages a pair was compared on, the reasons a pair is not compared on a page, and the number of pairs never compared with each other. For results from a version that did not record --min-chars, the card says that the rules for leaving a pair out are not recorded.
• Cut outputs: per model, the number of pages where the judge's copy of its output was cut at the text cap. For rows stored without the truncation flags the card says the count is not recorded.
• The link to the judge prompts uses the version that judged, when the metadata records it (judge: fix the display order of each comparison per page and pair #91).

New columns: rank_low, rank_high, truncated_pages in overall.jsonl; rank_low, rank_high in slices.jsonl. New board.json keys: pair_pages_min, pair_pages_max, pairs_never_compared, judged_with_version.

What the rank range does and does not say

Two reviewers simulated it independently (Bradley-Terry data with and without page effects; 3, 9 and 20 models; 10 to 300 pages; 300 to 400 boards per cell). Their numbers agree within 1.5 points.

Property	Result
A model's true rank is outside its range	at most 1.7% of models at 30 pages, at most 0.6% at 100 or more
Equally good pair called apart (each test is set to 5%)	6.3–7.4% at 20 pages, 5.7–6.9% at 30, 4.8–5.9% at 300; 4.3–4.7% on two-model boards; 8.4–9.9% at 10 pages, which is why the gate is 20
Unequal pairs separated (boards of 3 or more models)	the paired test is 4 to 16 points above "intervals do not overlap" in every cell
Wrong-direction calls	at most 0.8% of pairs

Limits:

• A range wider than one rank does not always show which models it is apart from. Two models with overlapping ranges can be clearly apart. The card says so.
• Two ranges that do not overlap do not prove that the pair is apart. "Clearly above" is not transitive: reviewers built boards where A is clearly above C and D, C and D are clearly above B, and A against B fails its own test. It did not occur in 18,894 non-overlapping pairs on 4,200 random boards. The card makes no claim either way.
• Each pair is tested on its own. In simulations, on a board of 9 equally good models, more than half of the boards had at least one pair that tested apart by chance. The card gives the per-pair rate.

Tests

• 59 new test cases; full suite 1112 passed; ruff, ty and rumdl pass.
• Rank range: clear separation, an unseparated pair, pairing (the same data with unpaired resamples does not separate), the level, exact twins, the 20-page gate on both models of a pair, a board where no pair can be tested, the replicate floor, the fitted-rank clamp alone and together with the gate, the non-transitive board, and "a range of one rank means apart from every other model" on random boards.
• Card: rows compared with compute_elo(...).rank_range, rendered cells, slice boards (including one too small for any pair), failed models, pages per pair in both storage orders, pairs never compared, cut outputs known and not recorded, the --min-chars wording for each case, the version filter, a test that compares the rank-range paragraph word for word, and a test that keeps the withdrawn sentences off the card.
• Reviewers ran 37 mutants in round 2 and 37 in round 3; the survivors of each round now have tests.

Review before this PR

Four rounds, each by Codex and by an independent agent. No round found a defect in the computation after round 1; rounds 2 to 4 were about sentences.

• Round 1 (first commit): the card's first wording misread a range as a list of unseparated models; the refactor changed the percentile arithmetic at non-default levels; rows without truncation flags showed as zero cut pages; the test was too eager on few pages; the tests passed with an unpaired implementation.
• Round 2: the reading rule "ranges that do not overlap are clearly apart" is false in rare cases; the card gave the wrong reason for the paired test; a 12-page slice showed a full range in every row; "each pair" was false with pairs never compared.
• Round 3: four more sentences were true for typical boards and false for some board. The paragraph was rewritten to hold only definitions, what follows from them, and measurements named as such.
• Round 4: both reviewers judged the paragraph sentence by sentence. Codex found all ten true, with "ranked models" to be said where a FAILED model is in the table. The agent found nine true and one false on a two-model board ("two ranges do not decide a pair"), now reworded. It confirmed 0 failures of "a range of one rank means apart from every other ranked model" in 1,772 one-rank ranges, and 0 narrowed ranges in 92,963 comparisons for "can only widen". The remaining points were drift in docstrings, the CHANGELOG and the agent skill, all applied.

The changes after round 4 are wording only. The rank-range paragraph is one constant, and a test compares it word for word.

Not in this PR

• _check_value accepts an allow-listed string in a numeric column. This is an existing pattern in the privacy check, not new here.
• A published list of the pairwise results (which would let a reader see which pairs are apart).

🤖 Generated with Claude Code