[MaxGhenis]

WIP. Stacked on #240 (v2-microcosm-batches).

scripts/confirmatory_run.py runs the v2 calculator-interface study's confirmatory stage as its draft pre-registration specifies, and writes the run records the confirmatory report checks before it scores anything. It runs the treatment harness unchanged: tool_pilot_argon.py and policybench here are 6b00a6f's, and the runner refuses to start if they differ.

Nothing here runs a model. The confirmatory run waits on the frozen pre-registration and Max's go.

What it does

• Lanes. Two per model. Households alternate between them in draw order. A lane runs a household's arms in the order drawn for it (stream order/<model>), then its reruns in passes, and only then decides its next household. The six models run at once.
• One engine pool per model. The pool starts once, with no features of its own. Each call carries its arm's features in the harness's call frame. In development, starting a pool per arm cost 2.8 to 3.6 minutes each time.
• Admission. Each model has a cap and a reserve per household under way. After one refusal, every later decision is a refusal. A decision waits while a pass's reruns are undecided, so the runner always knows whether a household has finished.
• The hard ceiling. Overall cap plus $25, counted over every model through the harness's Spend. When spend crosses it, no attempt starts, conversations under way end budget_stop, and every model's log gets a ceiling event. Admission checks the ceiling and takes its time under the ceiling's lock, so no household is decided after the crossing.
• The attempt deadline. A conversation still running an hour after it started (the longest development attempt took 714 s) is abandoned. Its attempt gets a harness-error record, AttemptDeadline, with the spend so far, and the model's lanes stop with cause deadline. The conversation ends budget_stop at its next turn; its spend after the deadline still counts toward the ceiling and goes to the run log as late-cost events. An attempt whose conversation raises keeps the spend it counted.
• Failures. Each attempt gets a class from its record and the host evidence, in the report's precedence, and an action from its class. The class lists hold every litellm subclass the harness groups by isinstance, explain traces count as calls, and a Timeout on the escalated request after a truncated turn is class model. An operator failure pauses the model's lanes until confirmatory_run.py resume, accepted only if written after the pause began. A harness failure stops them, and so does one inconclusive error that ends runs in more than five households. Replaced attempts are saved, and in-flight records fsynced, before their reruns.
• Stops name their cause: harness, inconclusive, deadline, pool, runner-error, interrupt or sampler (ceiling events have cause ceiling). An exception in the runner itself stops every model's lanes after settling and writing what had ended. The signal handler only notes the signal, and the main loop stops the lanes.
• Host log. A sample every 30 seconds, with the monotonic clock beside the wall clock, then the guard's and the power log's lines for the booking. A failed memory-pressure reading is no sample. A read of the guard's or the power log is tried five times before it is a runner error. A pass's classes wait until its host evidence is complete, which takes at most about 90 seconds. A sampler whose samples fail for five minutes, or whose thread ends, stops every model.
• Chains and anchors. Every line of the run, admission and host logs carries prev, the sha256 of the line before. Every 15 minutes and at the end, anchors.jsonl gets every log's line count and head and every manifest's sha256, and the design's anchor_command runs on it in the background. scripts/confirmatory_anchor.py is that command: it commits the file to a clone and pushes it, so a remote outside the run holds each anchor's time. Each push's result goes to anchor-pushes.jsonl.
• Workers. --workers must equal the design's engine workers per model.
• Logs from the start. Each model's run and admission logs exist from its manifest on, so a model that never decides (the ceiling crossed first) still has both.
• A clock that never goes back. The host's wall clock steps back by up to about 0.1 ms, even within a thread. The study's clock probe saw 29 backward steps in 177 million readings by twelve threads, the largest 99 µs, at a load of about 150. The run records order attempts, stops, admission decisions and the ceiling's crossing by their times, and an attempt's end and the next start can be microseconds apart. So clock() returns time.time(), never earlier than a time it returned before, and each such time is taken under the lock that orders it. The harness's request times stay raw. The report allows them 0.1 s of slack, widened by the largest backward step the host log shows.

Invariants

The tests' oracle (run_problems in tests/test_confirmatory_run.py) checks these on every fake run, and the property test checks them on random mixes of planted failures:

• Every attempt has one run-log line, and every record and replaced-attempt file hashes as its line logs it.
• Each logged class and action equals the one recomputed from the record and the host evidence.
• Each lane runs its households in draw order. Each household runs its drawn arm order, then its reruns in passes. No two attempts on a lane overlap. No arm has more than two reruns.
• Each admission decision equals the rule applied to the finished households' spend and the households under way. A household a stop cut short is under way until the stop. After one refusal, every decision refuses.
• clock() never returns a time earlier than one it has returned, including under a wall clock that steps back (test_the_clock_never_goes_back and test_a_wall_clock_that_steps_back_keeps_the_run_procedure). The run log itself is not in time order: an attempt's line is written when its household ends, and a ceiling event carries the crossing's time. Every check that orders events reads their times, never their line order.
• Every record's request times lie within its attempt, to within 1 s.
• No attempt starts during a pause, after a stop, or after the ceiling's crossing.
• Conservation: the ceiling's total equals the spend before the stage plus every record's cost plus every late-cost event.
• Each log is an unbroken chain from its seed; each anchor matches the logs' prefixes at its line counts; the last anchor matches every log whole, and its push returned 0.
• Every stop names a cause. Each harness, deadline and inconclusive trigger is followed by a stop of that cause within 5 s, no attempt starts after a harness or deadline trigger, and every stop of those causes has its trigger.

Tests

• tests/test_confirmatory_run.py and tests/test_confirmatory_anchor.py: 189 pass locally at f5099db (load about 90). f5099db makes the interrupt test independent of timing: it now interrupts from inside a provider call and waits for the lanes to stop, and it passed 5 of 5 at load 91. 19 mutants of the rules added in fdbd411 each fail a named test; the one that first survived (a resume kept until a pause) is now caught by a resume with a misdated time. Until fdbd411, 121 tests passed in 8 runs of the full suite in a row at loads from 12 to 146. They cover unit tests (streams, arms, classes, host evidence, the ceiling), fake runs with planted failures, the harness's own conversation under the runner, and the property test. Stacked PRs into v2-microcosm-batches get no CI, since ci.yml runs on pull requests to main.
  • Three oracle faults, fixed in 808ccc8. First, the oracle took a household that a stop cut short to have finished at its last attempt. The runner counts it as under way until the stop, and a new test makes that case deterministic. Second, under load the sampler stalled past the 90 ms scaled host-log gap, so an untouched run got host reruns. The gap is now 0.5 s in the tests. Third, the oracle now allows the harness's raw request times 1 s of slack.
  • The clock. A test reads the wall clock up to 0.2 ms early everywhere. Without clock() it fails every time: attempts overlap, and a household runs before its admission.
• Differential test, in the study's scripts repo (test_confirmatory_differential.py). It runs this runner's Run on a copy of the development board. The harness's run_household is real, and so are this runner's host log and lanes. Only three things are fakes: the model (replying in each API's shape: chat tool, chat JSON, Responses), the engine pool and the host's readers. The confirmatory report's analyse then checks every record and log. At the report's eda54e1 against f5099db, 22 tests pass, among them the same jittered clock, and a 60-example soak of its property test passed.
  • It found one defect here: the missing admission log, fixed in ec31e83.
  • It found four rules the report applied more strictly than the procedure. All four are fixed in the report:
    • an arm left empty by a stop;
    • a rerun cut off by a stop;
    • a household cut short by a stop, which counts as under way until the stop;
    • a truncated, retried request at the ceiling.
  • Six planted runner faults each fail the report, so the test can fail:
    • a skipped provider rerun;
    • admission past the cap;
    • a runner blind to memory pressure;
    • a runner that ignores a harness failure;
    • the ceiling's crossing logged an hour late;
    • a lane that stops deciding.

Plan

• Runner
• Tests
• Differential test (in the study's scripts repo): run this runner on fakes and check its records pass the confirmatory report
• Independent review, round 1 (Opus 5.5): REQUEST_CHANGES, nine findings, all answered in fdbd411
• Differential test updated for fdbd411's API (in the study's scripts repo)
• Independent review, round 2, running

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 26, 2026 2:28pm UTC