Skip to content

Commit 71d9052

Browse files
TheGreatAxiosTheGreatAxios
authored
feat(evals): add completion-rate baseline harness (#1032)
* feat(evals): add task-completion baseline harness for CL-7932 Summary: repeatable harness runs a frozen 4-task set end to end through the production agent loop with scripted mock responders, reporting completion rate plus control-layer signals (turns, retries, compaction events, doom-loop/thrash interventions, wall clock). Baseline on b509ed3: 50.0% (4/8 runs, 2 repeats), deterministic across repeats. Task set frozen as tasks.json v1 for the post-0.4.x re-measure. Verification: bun scripts/eval-completion.ts --repeats 2 --out evals/completion/baseline-2026-09-14.json; bun test evals/completion/lib.test.ts (7 pass); bun run check (exit 0). * fix(evals): make completion-harness withTimeout actually race the deadline The sync finally cleared the timer before Promise.race settled, so RunStatus timeout was unreachable and --timeout-ms was a no-op. Await the race so the timer clears on settle, export withTimeout for testing, and guard main() with import.meta.main. Adds scripts/eval-completion.test.ts proving a hung run rejects after the timeout. Baseline re-run still 50.0% (4/8). * fix(evals): keep timeout status, cancel in-flight work, aggregate gate suspensions (#1047) A run that times out after emitting a failure signal kept the timeout status instead of being recorded failed. The deadline now aborts the in-flight send and the runner quiesces the agent pump and collector before grading, so verify never runs on partial state. Gate suspensions are summed into completion totals and printed in the human summary instead of being collected and dropped. * fix(evals): contain completion-harness task paths and validate timeout (#1049) * fix(evals): contain completion-harness task paths and validate timeout Confine custom task-set fixture/script/verify references to the harness directory, mark the version-controlled grading trust boundary, reject degenerate timeout values, and re-record the baseline against the current tree. * fix(evals): align baseline and totals with gate-suspension signal Carry the gate-suspension total through schema, aggregation, and summary so the checked-in baseline validates. Add a regression test asserting the frozen baseline against the current report schema. * fix(evals): label estimated turns and failed tool calls honestly (#1062)
1 parent 1128ed7 commit 71d9052

18 files changed

Lines changed: 1614 additions & 0 deletions

File tree

‎evals/completion/baseline-2026-09-14.json‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
{
2+
"harness": "completion-baseline",
3+
"version": 1,
4+
"startedAt": "2026-09-14T20:11:37.538Z",
5+
"finishedAt": "2026-09-14T20:11:41.269Z",
6+
"commitSha": "b1bace7e4a6c469050df048d53e7d1dea4b89ab6",
7+
"provider": "stub-scripted",
8+
"model": "completion-baseline-v1",
9+
"repeats": 2,
10+
"taskSetVersion": 1,
11+
"taskIds": ["version-endpoint", "sum-fix", "decline-refactor", "stall-read"],
12+
"results": [
13+
{
14+
"taskId": "version-endpoint",
15+
"title": "Add GET /version to a two-route service",
16+
"profile": "solve",
17+
"repeat": 0,
18+
"completed": true,
19+
"runStatus": "completed",
20+
"turnsUsed": 3,
21+
"toolCallCount": 2,
22+
"failedToolCalls": 0,
23+
"retryCount": 0,
24+
"compactionEvents": 0,
25+
"doomLoopInterventions": 0,
26+
"thrashInterventions": 0,
27+
"gateSuspensions": 0,
28+
"agentDurationMs": 567,
29+
"verifyDurationMs": 196,
30+
"verifyExitCode": 0,
31+
"overBudget": false
32+
},
33+
{
34+
"taskId": "version-endpoint",
35+
"title": "Add GET /version to a two-route service",
36+
"profile": "solve",
37+
"repeat": 1,
38+
"completed": true,
39+
"runStatus": "completed",
40+
"turnsUsed": 3,
41+
"toolCallCount": 2,
42+
"failedToolCalls": 0,
43+
"retryCount": 0,
44+
"compactionEvents": 0,
45+
"doomLoopInterventions": 0,
46+
"thrashInterventions": 0,
47+
"gateSuspensions": 0,
48+
"agentDurationMs": 167,
49+
"verifyDurationMs": 151,
50+
"verifyExitCode": 0,
51+
"overBudget": false
52+
},
53+
{
54+
"taskId": "sum-fix",
55+
"title": "Fix an off-by-one in a total function",
56+
"profile": "solve",
57+
"repeat": 0,
58+
"completed": true,
59+
"runStatus": "completed",
60+
"turnsUsed": 3,
61+
"toolCallCount": 2,
62+
"failedToolCalls": 0,
63+
"retryCount": 0,
64+
"compactionEvents": 0,
65+
"doomLoopInterventions": 0,
66+
"thrashInterventions": 0,
67+
"gateSuspensions": 0,
68+
"agentDurationMs": 72,
69+
"verifyDurationMs": 188,
70+
"verifyExitCode": 0,
71+
"overBudget": false
72+
},
73+
{
74+
"taskId": "sum-fix",
75+
"title": "Fix an off-by-one in a total function",
76+
"profile": "solve",
77+
"repeat": 1,
78+
"completed": true,
79+
"runStatus": "completed",
80+
"turnsUsed": 3,
81+
"toolCallCount": 2,
82+
"failedToolCalls": 0,
83+
"retryCount": 0,
84+
"compactionEvents": 0,
85+
"doomLoopInterventions": 0,
86+
"thrashInterventions": 0,
87+
"gateSuspensions": 0,
88+
"agentDurationMs": 68,
89+
"verifyDurationMs": 164,
90+
"verifyExitCode": 0,
91+
"overBudget": false
92+
},
93+
{
94+
"taskId": "decline-refactor",
95+
"title": "Decline a migration that does not fit the turn budget",
96+
"profile": "decline",
97+
"repeat": 0,
98+
"completed": false,
99+
"runStatus": "completed",
100+
"turnsUsed": 1,
101+
"toolCallCount": 0,
102+
"failedToolCalls": 0,
103+
"retryCount": 0,
104+
"compactionEvents": 0,
105+
"doomLoopInterventions": 0,
106+
"thrashInterventions": 0,
107+
"gateSuspensions": 0,
108+
"agentDurationMs": 25,
109+
"verifyDurationMs": 99,
110+
"verifyExitCode": 1,
111+
"overBudget": false
112+
},
113+
{
114+
"taskId": "decline-refactor",
115+
"title": "Decline a migration that does not fit the turn budget",
116+
"profile": "decline",
117+
"repeat": 1,
118+
"completed": false,
119+
"runStatus": "completed",
120+
"turnsUsed": 1,
121+
"toolCallCount": 0,
122+
"failedToolCalls": 0,
123+
"retryCount": 0,
124+
"compactionEvents": 0,
125+
"doomLoopInterventions": 0,
126+
"thrashInterventions": 0,
127+
"gateSuspensions": 0,
128+
"agentDurationMs": 27,
129+
"verifyDurationMs": 86,
130+
"verifyExitCode": 1,
131+
"overBudget": false
132+
},
133+
{
134+
"taskId": "stall-read",
135+
"title": "Stall by re-reading the same file instead of acting",
136+
"profile": "stall",
137+
"repeat": 0,
138+
"completed": false,
139+
"runStatus": "failed",
140+
"turnsUsed": 3,
141+
"toolCallCount": 3,
142+
"failedToolCalls": 0,
143+
"retryCount": 0,
144+
"compactionEvents": 0,
145+
"doomLoopInterventions": 1,
146+
"thrashInterventions": 0,
147+
"gateSuspensions": 0,
148+
"agentDurationMs": 318,
149+
"verifyDurationMs": 30,
150+
"verifyExitCode": 2,
151+
"overBudget": false,
152+
"error": "reactor error: Doom loop detected: an identical tool batch (read_file) executed 3 times consecutively"
153+
},
154+
{
155+
"taskId": "stall-read",
156+
"title": "Stall by re-reading the same file instead of acting",
157+
"profile": "stall",
158+
"repeat": 1,
159+
"completed": false,
160+
"runStatus": "failed",
161+
"turnsUsed": 3,
162+
"toolCallCount": 3,
163+
"failedToolCalls": 0,
164+
"retryCount": 0,
165+
"compactionEvents": 0,
166+
"doomLoopInterventions": 1,
167+
"thrashInterventions": 0,
168+
"gateSuspensions": 0,
169+
"agentDurationMs": 332,
170+
"verifyDurationMs": 52,
171+
"verifyExitCode": 2,
172+
"overBudget": false,
173+
"error": "reactor error: Doom loop detected: an identical tool batch (read_file) executed 3 times consecutively"
174+
}
175+
],
176+
"totals": {
177+
"tasksTotal": 4,
178+
"runsTotal": 8,
179+
"completedRuns": 4,
180+
"completionRate": 0.5,
181+
"meanTurnsToCompletion": 3,
182+
"meanAgentDurationMs": 197,
183+
"totalRetries": 0,
184+
"totalCompactionEvents": 0,
185+
"totalDoomLoopInterventions": 2,
186+
"totalThrashInterventions": 0,
187+
"totalGateSuspensions": 0
188+
}
189+
}

0 commit comments

Comments
 (0)