What
The guard at apps/framework/harness/run-eval.ts throws only when stoppedReason === 'error' and the transcript is empty. A container killed mid-flight arrives with a populated transcript and a plausible partial score, so it is scored, written and publishable.
One did on 21 August: stoppedReason: error_exit_255, 15 tool calls, written as a 2/6 agent failure on benchmark-outpost-004-queue-destination. It was caught only because a baseline failing after three passes looked odd enough to open.
It is the 13 August incident in a new costume — there, dead runs were identified by fingerprint and four were missed because a negative check passing made them look genuine. The same shape recurs here: the two "passing" checks were the negative ones an idle agent satisfies.
Why the new triage script is not the fix
triage.ts flags a non-clean stoppedReason at read time, which is useful and not a guard. The row is still scored, still written, and still eligible for a snapshot. A flag depends on someone reading it; the existing empty-transcript guard does not.
Done when
A non-clean stoppedReason is excluded from scoring the way an empty transcript already is — the cell errors and is re-runnable, rather than producing a number. Worth checking the full set of terminal reasons the agent runners emit before choosing the allow-list, since stop, end_turn and complete are all in use.
What
The guard at
apps/framework/harness/run-eval.tsthrows only whenstoppedReason === 'error'and the transcript is empty. A container killed mid-flight arrives with a populated transcript and a plausible partial score, so it is scored, written and publishable.One did on 21 August:
stoppedReason: error_exit_255, 15 tool calls, written as a 2/6 agent failure onbenchmark-outpost-004-queue-destination. It was caught only because a baseline failing after three passes looked odd enough to open.It is the 13 August incident in a new costume — there, dead runs were identified by fingerprint and four were missed because a negative check passing made them look genuine. The same shape recurs here: the two "passing" checks were the negative ones an idle agent satisfies.
Why the new triage script is not the fix
triage.tsflags a non-cleanstoppedReasonat read time, which is useful and not a guard. The row is still scored, still written, and still eligible for a snapshot. A flag depends on someone reading it; the existing empty-transcript guard does not.Done when
A non-clean
stoppedReasonis excluded from scoring the way an empty transcript already is — the cell errors and is re-runnable, rather than producing a number. Worth checking the full set of terminal reasons the agent runners emit before choosing the allow-list, sincestop,end_turnandcompleteare all in use.