Correctness sweep: dead processes, wildcards, an inert field, and --resume - #55
Merged
Conversation
…esume Four separate faults, none of which needed a run to find. **A dead process is no longer scored.** The guard only rejected a run that errored *and* left nothing, so a container killed mid-flight came through with a populated transcript and a plausible partial score. One did on 21 August — error_exit_255, fifteen tool calls, written as a 2/6 agent failure — and was spotted only because a baseline failing after three passes looked odd. Budget exhaustion stays scoreable: timeout, max_steps and error_max_turns all mean the agent had its run and used it. A deny-list rather than an allow-list, because runners derive stop reasons from three different sources and an uncatalogued one is likelier to be a new legitimate outcome than a new crash. Caveat recorded in the code: exit 137 maps to `timeout`, and 137 is also what a stopping Docker daemon sends. **Wildcard topics, decided per site rather than blanket.** Rejected for outpost-004's queue, where "they only want orders on the queue" means a wildcard puts the customer's retry traffic there too — the acted-too-broadly failure that scenario is built around, and applyOutpostSeed's own default. Rejected for outpost-003's secondary alert, where accepting it let an agent pass without ever finding either topic name. Accepted, and now documented as intentional, for outpost-003's primary check and outpost-005's platform-fault guard: both ask whether anything would arrive, and a wildcard means yes. **tenants[].topics removed** from the schema and four seed files. Verified live that the API ignores it; it had already produced a scorer justifying its design on an agent "editing the tenant's topics", a route that does not exist. **--resume added; the re-run default kept.** A stale row surviving a code change is a live hazard — outpost-001's scorer, prompt and seed were all corrected on 24 August, and keeping its old rows would have published results scored by code that no longer exists. What was missing was a way to resume after an interruption, which three runs needed this week. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
Those twelve rows were scored by code that no longer exists: both scenarios stopped accepting a wildcard where a specific topic is the point. Keeping them would have published results produced by a scorer that had since changed, which is the reason outpost-001 was re-run for v0.3.0. 11 of 12 pass. The tightening cost nothing — every agent that would have passed still passes, because they were naming topics explicitly rather than subscribing to everything. The wildcard holes were routes nobody had taken yet, which is the good version of that finding. The single failure is outpost-003's familiar variance: the agent found /operator-events, hit the rejected PATCH /config, and stopped to ask for input rather than continuing to enumerate — the same mode as one of the three majority passes yesterday. 104 -> 103. compare-snapshots flags it as not attributable to the agent because that scenario's scorer changed between refs, which is the correct default even though the actual cause was variance. The published page is unaffected: it reads from the v0.3.0 tag, so this snapshot is the basis for the next release rather than a change to that one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four faults from the scorer audit, none of which needed a run to find. Plus the twelve cells they invalidated, re-measured.
A dead process is no longer scored — #45
The guard only rejected a run that errored and left nothing behind, so a container killed mid-flight came through with a populated transcript and a plausible partial score. One did on 21 August —
error_exit_255, fifteen tool calls, written as a 2/6 agent failure — and was spotted only because a baseline failing after three passes looked odd.Budget exhaustion stays scoreable:
timeout,max_stepsanderror_max_turnsall mean the agent had its run and used it up.A deny-list rather than an allow-list, because runners derive stop reasons from three different sources and an uncatalogued one is likelier to be a new legitimate outcome than a new kind of crash.
Caveat recorded in the code: exit 137 maps to
timeout, and 137 is also what a stopping Docker daemon sends. Telling those apart needs the harness's elapsed time, which that layer doesn't have.Wildcard topics, decided per site — #48
*accepted?004queueorderson the queue" — a wildcard puts the customer's retry traffic there too, the acted-too-broadly failure the scenario is built around003secondary alert003primary alert005platform guardtenants[].topicsremoved — #49Verified live that the API ignores it: a
PUTcarrying topics returns[]. It had already produced a scorer justifying its design on an agent "editing the tenant's topics", a route that does not exist. Gone from the schema and four seed files.--resumeadded, re-run default kept — #50A stale row surviving a code change is a live hazard — this PR is itself an instance, since
003and004needed re-running. What was missing was a way to resume after an interruption, which three runs needed this week.The re-measurement
11 of 12 pass. The tightening cost nothing: every agent that would have passed still passes, because they were naming topics explicitly. The holes were routes nobody had taken yet.
The one failure is
003's familiar variance — found/operator-events, hit the rejectedPATCH /config, stopped to ask for input.compare-snapshots: 104 → 103, flagged as not attributable to the agent because that scorer changed. Correct default, even though the real cause was variance.The published page is unaffected — it reads from the
v0.3.0tag, so this snapshot is the basis for the next release.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA