Leave recorded ceiling-exhaustion rows alone when applying an adjudication

[MaxGhenis]

What changes

apply_adjudications rewrites every row of an adjudicated case except parse_contract_failure rows. So it overwrote a recorded budget_exhausted_at_ceiling row with the adjudicated class and subtype. Elsewhere the pipeline treats that recorded class as authoritative:

• collect_audit (_row_failure_source and the deterministic branch) and case_annotations.wrong_prediction_rows force it onto its row.
• full_run_export.merge_annotations restores it at dashboard export. It restores only failure_source, so the export kept the overwritten subtype.

The annotation CSVs kept the overwrite.

• New NON_SUBSTANTIVE_SOURCES (parse_contract_failure, budget_exhausted_at_ceiling): the row classes an adjudication leaves alone. apply_adjudications skips both.
• verify_adjudications_applied uses the same mask for its substantive rows, which must carry the adjudicated source and subtype. It also uses it for its stray-verdict check: no substantive row may still carry the judge's non-final class.
  • Without the first, the release drivers' apply-then-verify (scripts/finish_musewonder.py triage, finish_haiku55.py ~2901–2904) would demand the adjudicated class on a recorded ceiling row that apply now leaves alone.
  • Without the second, a case whose judge also chose budget_exhausted_at_ceiling would fail verification on its recorded rows.
  • For parse_contract_failure rows the stray check is unchanged: a final class never matches a non-final judge class.

Background: ~/reviews/policybench-budget-line-20261013/REPORT.md ("How the label is assigned") and sol_review_r3.md finding 3.

No published effect

• No committed row or case note records budget_exhausted_at_ceiling. There are 10,346 rows and 989 case notes at origin/main (release 20261013, bc30579) and at Re-judge five audit verdicts the judge-isolation audit could not clear (release dashboard-data-20261013b) #218's 20261013b (6c44858). No case in either release's frozen audit manifest records the class, and no prediction row in the Gemini 4 Argon release stage (Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221) records it.
• Rebuild: I rebuilt the row annotations and case notes as triage builds them: collect_audit on the frozen audit tree, the committed adjudications, the release's wording amendments. On this branch both releases' CSVs come out byte-identical to the committed files (sha256 in ~/reviews/policybench-budget-line-20261013/pr-b-impact/sha256.txt; scripts in ../pr-a-impact/).
• Differential test: test_skipping_ceiling_rows_moves_no_committed_annotation applies the committed record to the committed annotations under the old mask and the new one, and checks the results are identical.

Invariants

These are stated and tested with Hypothesis in test_adjudication_skips_non_substantive_rows_properties. It runs 300 generated examples: one to three cases of one to four rows, each of any class and subtype (skipped classes oversampled), and a valid record (checked by parse_adjudications) deciding any subset of cases from any judge class.

1. A parse_contract_failure or budget_exhausted_at_ceiling row keeps its class and subtype, and a row of an undecided case is untouched.
2. Every other row of a decided case takes the adjudicated class and subtype.
3. The report's rows_rewritten counts exactly the rows whose class or subtype changed.
4. Round trip: verify_adjudications_applied accepts what apply_adjudications builds.
5. Idempotence: a second application changes nothing.

The property's oracle spells out the skipped classes itself instead of importing the constant. On its own it catches each of four mutants:

• apply's mask reverted;
• verify's substantive mask reverted;
• the stray check run over every row;
• a substantive class added to the skip list.

Tests

• tests/test_adjudications.py:
  • test_apply_leaves_a_recorded_ceiling_exhaustion_alone;
  • test_verify_does_not_hold_a_ceiling_row_to_the_adjudicated_class, which also checks that rewritten rows are still held to the record;
  • the differential test and the property above.
• Locally: pytest passes on test_adjudications and the release-driver suites (results below). ruff check . and ruff format --check . are clean.

Not changed

• No change forces a recorded ceiling class onto the case note in collect_audit's judged branch. eval_no_tools.py attaches the label only to missing variables, so a case whose wrong models all record it is parse_failure_only and already takes the deterministic branch.
• No frozen audit tree or release artifact is edited. This reaches publication only through a future release driver.

axiom: n/a: benchmark annotation pipeline, no policy rule

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 13, 2026 11:41pm UTC