fix(evals): keep timeout status, cancel in-flight work, aggregate gate suspensions - #1047
Merged
TheGreatAxios merged 1 commit intoSep 15, 2026
Conversation
…e suspensions A run that times out after emitting a failure signal kept the timeout status instead of being recorded failed. The deadline now aborts the in-flight send and the runner quiesces the agent pump and collector before grading, so verify never runs on partial state. Gate suspensions are summed into completion totals and printed in the human summary instead of being collected and dropped.
TheGreatAxios
merged commit Sep 15, 2026
4827e63
into
cl-7932-baseline-task-completion-rate-before-the-04x-line-and-re
13 checks passed
TheGreatAxios
added a commit
that referenced
this pull request
Sep 15, 2026
* feat(evals): add task-completion baseline harness for CL-7932 Summary: repeatable harness runs a frozen 4-task set end to end through the production agent loop with scripted mock responders, reporting completion rate plus control-layer signals (turns, retries, compaction events, doom-loop/thrash interventions, wall clock). Baseline on b509ed3: 50.0% (4/8 runs, 2 repeats), deterministic across repeats. Task set frozen as tasks.json v1 for the post-0.4.x re-measure. Verification: bun scripts/eval-completion.ts --repeats 2 --out evals/completion/baseline-2026-09-14.json; bun test evals/completion/lib.test.ts (7 pass); bun run check (exit 0). * fix(evals): make completion-harness withTimeout actually race the deadline The sync finally cleared the timer before Promise.race settled, so RunStatus timeout was unreachable and --timeout-ms was a no-op. Await the race so the timer clears on settle, export withTimeout for testing, and guard main() with import.meta.main. Adds scripts/eval-completion.test.ts proving a hung run rejects after the timeout. Baseline re-run still 50.0% (4/8). * fix(evals): keep timeout status, cancel in-flight work, aggregate gate suspensions (#1047) A run that times out after emitting a failure signal kept the timeout status instead of being recorded failed. The deadline now aborts the in-flight send and the runner quiesces the agent pump and collector before grading, so verify never runs on partial state. Gate suspensions are summed into completion totals and printed in the human summary instead of being collected and dropped. * fix(evals): contain completion-harness task paths and validate timeout (#1049) * fix(evals): contain completion-harness task paths and validate timeout Confine custom task-set fixture/script/verify references to the harness directory, mark the version-controlled grading trust boundary, reject degenerate timeout values, and re-record the baseline against the current tree. * fix(evals): align baseline and totals with gate-suspension signal Carry the gate-suspension total through schema, aggregation, and summary so the checked-in baseline validates. Add a regression test asserting the frozen baseline against the current report schema. * fix(evals): label estimated turns and failed tool calls honestly (#1062)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Fixes CL-7968