Skip to content

scoreboard: arm 7, the code-as-action agent's first run - #1307

Merged
abrichr merged 12 commits into
mainfrom
claude/scoreboard-arm7
Oct 19, 2026
Merged

abrichr merged 12 commits into
mainfrom
claude/scoreboard-arm7

Conversation

@abrichr

@abrichr abrichr commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Adds arm 7 (code_agent) to the scoreboard runner, per PROTOCOL.md. Code only: the rows are run by the scoreboard's single runner under the load gate.

python -m scoreboard.run code_agent <task> --run-id <id> --n N runs one first run of ClaudeAgent(vision="code") (claude-sonnet-5-5, effort medium) per item, inside the compile-agent harness and save gate. Each run is recorded by the same JIT wrapper as arm 2, read back from the system of record, compiled, checked against the agent's trace (check_trace_against_ir), and taken through the same _admission(). It uses add_spend(), schema.Row, schema.timed() and write_meta(model=...) unchanged. OpenEMR uses tasks/openemr.py (item_values, check, oracle_verifier, setup, entry_url). Arms 1 to 3 are unchanged; the two JIT builders take an optional dialect.

Merge order: #1277, then #1305 (the agent itself), then this. Until those land, this diff also shows their commits (this branch has #1305 merged in).

Trial runs set aside (superseded/, not in the scoreboard, untimed, $0.31 on the scoreboard key, not yet in raw/spend.json): MockMed 5 of 5 correct and admitted; OpenEMR items 701 to 705: 4 correct and 1 safe stop on hand review, with admission refused for all 5 ("a write step lacks an independent read-back": no system-of-record change bound to the confirm-dialog save step); 5 earlier OpenEMR runs refused at the first drop-down by a gate rule since fixed in #1305. Arm 7's OpenEMR items now start at 721, clear of the four charts the trial saved.

public-artifacts.json: the inventory writer also registered this branch's existing raw files (arms 1 to 3), which were not listed yet.

🤖 Generated with Claude Code

Base automatically changed from claude/launch-demo-cua-to-program-8fa902 to main October 19, 2026 15:21
abrichr and others added 2 commits October 19, 2026 14:41
…, keep the JIT cache outside the source tree

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr merged commit 4e31711 into main Oct 19, 2026
1 check passed
@abrichr
abrichr deleted the claude/scoreboard-arm7 branch October 19, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant