Current next action
PR-B: run the 340 measured episodes — awaiting a decision on how to pace it.
Every prerequisite is on main (#857, #859). measured_run_allowed is still
false; flipping it is what starts the run, and the run's duration is dominated by
something nobody has measured.
measured AOS ~40s of fixed cost per episode, gitseed ~5s → ~1h total
unmeasured agent time per episode, and 1,020 judgements
bounded only by the 30-minute per-episode timeout → worst case ~85h at 2 workers
The agent time cannot be measured without running a benchmark episode, which §33
forbids as a pilot. So the honest position is: the floor is about an hour, the
ceiling is days, and the real figure is unknown.
What the readiness work found
Seven defects in the pre-execution harness, plus one incident of my own making.
Five of the seven would have put a silent zero in the data:
| defect |
what would have been zero |
| suppression removed nothing for one candidate |
20 episodes, a manipulation that never happened |
| regression scored by exit code against an already-red suite |
all 160 AOS episodes |
four acceptance commands not runnable (pytest absent, one prose) |
80 episodes across both repositories |
| a pre-start failure recorded as an episode outcome |
any infrastructure failure |
| regression measured after the acceptance was installed |
all 160 AOS episodes, surviving the baseline fix |
Two more made judgements misstate what they judged: every judgement would have
reported its packet id as work.judge-1, and every packet directory was named
packet, so all 340 results would have collided in one folder.
None of these could have been caught by CI or by a unit test — each component did
exactly what it was told. They were found by asking whether each path can produce a
correct answer at all, with an input whose answer is known.
Verified end to end
suppression 17/17 on real trees, incl. 8 candidates with prefix-colliding ids
regression baseline frozen, stable across 3 runs, fails on a new break
acceptance 17/17 runnable and failing on base, as the protocol requires
scoring path produces a pass in both repositories
delivery 17/17 ON payloads carry the target decision
FVR feasibility 13/17 tasks can carry a functionally passing violation
runner 28/28 end to end on a real assignment, both arms
judge path a real judgement, correct label, correct packet id
analysis 6/6 scenarios, 33 controls, 21/21 mutations caught
two analysts 11/12 deterministic quantities matched exactly
Incident
Two of the 340 assignments had a first model contact outside the measured run,
because I tested that the batch runner starts and starting is the run. No row, no
saved events, zero files modified. Recorded in
incidents/2026-08-28-accidental-episode-start; batch.py --plan now exists so
checking no longer starts.
Not claimed
No measured episode has run. Nothing on main is evidence about the product's
effect.
Current next action
PR-B: run the 340 measured episodes — awaiting a decision on how to pace it.
Every prerequisite is on
main(#857, #859).measured_run_allowedis stillfalse; flipping it is what starts the run, and the run's duration is dominated bysomething nobody has measured.
The agent time cannot be measured without running a benchmark episode, which §33
forbids as a pilot. So the honest position is: the floor is about an hour, the
ceiling is days, and the real figure is unknown.
What the readiness work found
Seven defects in the pre-execution harness, plus one incident of my own making.
Five of the seven would have put a silent zero in the data:
pytestabsent, one prose)Two more made judgements misstate what they judged: every judgement would have
reported its packet id as
work.judge-1, and every packet directory was namedpacket, so all 340 results would have collided in one folder.None of these could have been caught by CI or by a unit test — each component did
exactly what it was told. They were found by asking whether each path can produce a
correct answer at all, with an input whose answer is known.
Verified end to end
Incident
Two of the 340 assignments had a first model contact outside the measured run,
because I tested that the batch runner starts and starting is the run. No row, no
saved events, zero files modified. Recorded in
incidents/2026-08-28-accidental-episode-start;batch.py --plannow exists sochecking no longer starts.
Not claimed
No measured episode has run. Nothing on
mainis evidence about the product'seffect.