[abrichr]

What

Code as action is now the default agent for a task's first run, and every save in that run is read back from the system of record before the agent goes on. A successful run compiles into a program as before.

• Default agent. ClaudeAgent() and openadapt-flow compile-agent now default to vision="code" on claude-sonnet-5-5, the setup scoreboard arm 7 measured. selective and every_turn remain available; selective and every_turn keep claude-opus-5-5.

• Read-back on every save the gate detects. After each action the save gate lets through, the harness reads the verify.yaml system of record and compares it with the read taken at the start of run 1 and with a read just before the save (harness.RecordReadBack, in both harnesses):

  • only a record created during run 1 can be the named one; a new record the task does not name, or any change to or removal of a starting record, stops run 1 (readback_wrong_record) after the wrong write has landed, and nothing can be admitted;
  • a second named record stops run 1 as a duplicate, counted even without record ids;
  • no new record is reported as "unconfirmed ... Do not save again", and the next save is refused if the first landed late;
  • a failed read stops run 1, and run 1 does not start without a starting read;
  • a save that raises is still read back;
  • landed requires every input value in the new record.

  Every read-back is kept in run1["readbacks"]. The pipeline always wires it.

• Teach path. A new MCP tool, teach_run (web apps), lets the client's model send action scripts. When a person approves saves (--teach-hold), the script stops before each save with needs_person, and the client's form asks the person. The approval is bound to that line on the app as it is at that moment; a stale approval does nothing. teach_act is unchanged.

• Recorder. After a save the gate let through, the recorder's after-read now waits for the system of record to change (Recorder.expect_write), so the compiler binds a late write to its save step. On OpenEMR the chart lands after the screen settles.

Measured (scoreboard, python -m scoreboard.build; PROTOCOL Amendments 10 to 12)

MockMed items 21 to 40 (20 items: item 30 is a save the server drops while the screen says saved, item 40 a save that lands on another patient). claude-sonnet-5-5. All rows are untimed because the host's load was above 12.

final attempt	Agent first runs (code, read-back)	Compiled program (from item 21)
Normal items correct	18 / 18	18 / 18
Item 30, dropped save	silent_wrong by the word rule; a safe stop on hand review	safe_stop
Item 40, wrong patient	flagged_wrong (the save landed on the other patient; the read-back then stopped run 1)	flagged_wrong
Silent wrong saves	1 by the scoreboard's word rule, 0 on hand review	0
Cost per item	$0.0093	$0
Model calls per item (median)	4	0
Task seconds, correct items (median)	6.6	5.3

On item 30 the agent said the save "did not take effect" because the read-back showed nothing was saved, but the scoreboard's word rule missed that wording. The row keeps the rule's verdict, and HAND_REVIEW.md records the review. The read-back's rules changed after review; the measured runs used the earlier rules, and by the new rules their verdicts are the same (Amendment 13; not re-run). From Amendment 11 on, arm 7 reports success only when its last read-back landed.

Real app (OpenEMR referral, local fixture). 6 of 6 correct by SQL read-back, and each save was read back as landed on the named chart. Admission still refuses these programs. The recorder fix binds the confirm save, but the "Create New Patient" click before it is write-shaped and writes nothing, so it has no read-back (Amendment 12). Fixing that changes what admission accepts, so it is not in this PR.

Spend: $0.43 on the scoreboard key (scoreboard total $12.98 of $30).

Tests

• tests/e2e/test_code_agent_default_e2e.py, MockMed in a real browser, scripted agents, no model calls, added to the flow-critical CI job:
  • read-back on every save;
  • a wrong-patient save stopped at the read-back;
  • a dropped save reported as not saved;
  • teach_run drafting a program;
  • teach_run stopping before a save and taking exactly one bound approval.
• Unit tests for record_readback, the new defaults, and the flags.
• 437 tests pass locally across the code-action, compile-agent, teach, JIT, authoring and recorder suites.

🤖 Generated with Claude Code

[abrichr]

AI review (coordinator)

Verdict: fix first. The direction is right and the e2e tests are real: MockMed in a real browser against the fault server, with no fakes standing in for the system of record. But the read-back accepts several writes it should refuse, and it can tell the agent "nothing was saved" when something was. Line numbers below are at fe7df6e91.

I checked RecordReadBack directly (unit probe, no browser) at this head:

case	outcome today	should be
a record that matches expect and existed before run 1 is overwritten	landed	refused
that record is deleted and a new matching one created	landed	refused
a second save creates a duplicate of the named record (keyed store)	landed	refused / held
the same duplicate in a store without an id field	no_change ("nothing was saved")	refused
mark() fails, then the second save is dropped	landed (the first save's write is counted)	unreadable, stop
the named record lands with an empty note	landed	landed_incomplete or similar

Tests run (serially, no xdist, host load 15 to 29): tests/test_code_action.py and tests/test_compile_agent.py pass (108 tests). All 5 tests in tests/e2e/test_code_agent_default_e2e.py pass, but test_teach_run_is_offered_and_drafts_a_program failed once under load with run 1 did not start: the app did not open in time and passed on rerun. This file is now in the flow-critical job, so expect that flake in the merge queue.

Blocking

1. A record that existed before run 1 counts as "the named record", so overwriting or deleting it reads back as landed. packages/flow/openadapt_flow/compile_agent/harness.py:127-144 (_others). named() matches only the expect identity fields (for example patient_id=p1, type=Triage). A change whose before and after both match, or a removal paired with any matching create, is allowed. On a real chart that already has a Triage record, run 1 can overwrite or replace clinical data, and the agent is told the save landed. check_run_record refuses it later, but by then the write is already in the EMR.
   Fix: for a create task, the named record is a record created during run 1 (its id or whole value is not in baseline()). Any change to or removal of a record that is in the baseline is wrong_record (or a new overwrote_existing), and it stops run 1. Add an e2e test that seeds a matching record before run 1.

2. A duplicate save is never caught, and in a store without an id field it reads back as "nothing was saved". harness.py:146-170. _others allows any number of named records to be created, so a second save on the named record reads back as landed. With no id field, diff_snapshots (record_sor.py:238-244) compares sets of serialized records, so an identical second row disappears. The agent is then told "nothing was saved ... do not report it as saved", which invites a third attempt.
   Fix: compare multisets (collections.Counter of _whole(r)) in the no-id path. In RecordReadBack, stop run 1 when more than one named record exists since baseline (duplicate), the same way wrong_record stops it.

3. A save that was sent but not confirmed can be retried blindly. harness.py:222 (readback_wait_s = 3.0, lines 337 and 778) and the no_change message at code_action.py:1503. A write that lands after 3 s reads back as no_change. The message says only "do not report it as saved", and nothing stops another save. Neither _CODE_SYSTEM (agents.py) nor the teach_run description tells the agent not to save again. In item 30 the agent happened to stop. Nothing enforces that.
   Fix: after a no_change or unreadable on a save, refuse the next gated commit unless a fresh read still matches the read taken at the no_change. If the store changed in between (a late landing), stop run 1 with RECONCILIATION_REQUIRED-style wording. Change the message to "the save is unconfirmed; do not save again; finish and report it as not confirmed".

4. A save whose dispatch raised is not read back. harness.py:470-481 and harness.py:891-899. On computer.execute raising (for example a Playwright timeout on a submit that navigates, which is common), the except BaseException re-raises before _read_back. In code_action a LookupError/UnsupportedAction then becomes "error: ... (a write may have been committed)" and the agent can resend the line. That is the "sent but not confirmed" case with no proof at all.
   Fix: when commit is not None, run _read_back in a finally (catching its own errors), log the outcome, and surface it in the error text. Treat no_change there like item 3 (no retry without a fresh matching read).

5. An unreadable system of record lets run 1 carry on. harness.py:150-151 returns unreadable, and _read_back (harness.py:213-235) only stops on wrong_record. If the baseline read failed (self._baseline is None, pipeline.py:513), every read-back is unreadable and the agent can make any number of saves with no wrong-record check. The PR says "every save ... is read back ... before the agent goes on". Fail closed instead.
   Fix: raise ReadBackRefused (status readback_unreadable) on unreadable. Also refuse to start run 1 when the baseline read is None. Separately, mark() swallows its exception and falls back to start (harness.py:121-125, 157). Then a dropped second save is reported landed on the strength of the first save. On a failed mark(), return unreadable and do not fall back.

6. Public copy says more than the code does. CAPABILITIES.md and llms-full.txt say "every save in that run is read back from the system of record before the agent goes on" and "the wrong-patient save stopped by the read-back". Today only actions the gate classifies as commits are read back, an unreadable read-back does not stop anything (item 5), and the item 40 write did land on another patient: run 1 stopped after it. They also say "no silent wrong save on hand review" while the generated SCOREBOARD.md row for arm 7 shows silent_wrong 1.
   Fix: after 1 to 5 land, say "every save the gate detects is read back; a save on another record is detected and run 1 stops". Quote the scoreboard row as generated and link HAND_REVIEW.md for item 30, rather than restating the hand verdict as the result.

Non-blocking

• Writes the gate does not detect get no read-back. A JS autosave, a link styled as a button, a select that submits without the submits flag, or Enter in a field all go unchecked until the next gated save. If none follows, they go unchecked entirely. When run 1 ends early (result.status != draft), check_run_record never runs, so a stray write is not reported at all. Do one final readback() at the end of run 1 regardless of status, and map wrong_record there to readback_wrong_record.
• The poll stops at the first change. harness.py:222 breaks on the first non-no_change read. A save that writes the named record first and another record 0.5 s later reads back as landed. The final record check catches that before admission, but one more read after a short settle would catch it in run 1.
• landed checks only that the named record changed, not the values. It does not check that the input values are in the record (see the empty-note row). Amendment 11 counts arm 7 as successful on landed alone. Compare against values the way check_run_record does, and report landed_partial.
• The approval binding (driven.py:159-164) does not capture the app state. It binds run id, action count, first line and control name. A page that changes without an agent action (timer, redirect) still matches. The web identity gate still runs after approval, so this is a weak spot, not a hole. Include the observation's text hash or the target element's scope.
• The MCP approval path for teach_run is not tested end to end. The e2e test calls teach.hold_needed and dispatch(held_by_person=...) directly. The mcp.py:395-410 branch (_confirm_with_person for teach_run) is not exercised.
• teach_run gives any MCP client the code tool's sh. On Linux without unprivileged user namespaces, the sandbox falls back to the flag allow-list only (sandbox.py:380-400). The sandbox itself is unchanged and its path checks look sound (absolute paths, .. and long options are refused; no flag that takes a value accepts a path). Report shell_isolation in the teach_start answer so a client knows which fence is in place.
• Each save polls a full store read about 12 times (0.25 s for 3 s) when nothing changes, plus mark(). That is fine for MockMed, but on a real EMR table, and on write-shaped clicks that write nothing (OpenEMR "Create New Patient"), it adds about 3 s and a dozen full reads. Consider a filtered read or a backoff.
• Scoreboard provenance. raw/openemr_referral/code_agent/2026-10-25-rb-admit/meta.json records engine bedf5439, but Amendment 12 says item 6 ran after the recorder fix, which is in 7525b685. Record a dirty-tree flag or diff hash in meta.json so rows can be reproduced from the commit they name.
• Local paths in committed data. meta.json files, scoreboard.json and ledgers/item-040.json contain /Users/abrichr/... paths. That is harmless, but strip them in the runner.
• Unrelated row in llms-full.txt. It gains a flow: induce search-open-act-return and row-varying loops, each row checked on its own record #1435 worklist row that is not part of this change, probably a regeneration. Check that it matches CAPABILITIES.md on main.
• The scoreboard read-back for MockMed hard-codes patient_id=p1 and an empty baseline (benchmark/scoreboard/run.py:759-763). This is correct for the current item set. Add a comment, or take both from the item, so a future item on another patient does not read as wrong_record.

Checked and fine

• Model calls: no model call on healthy compiled runs. The PR does not touch the replay path. Recorder.expect_write is called only from the run-1 harness.
• Licensing and size: no GPL/AGPL material. No vendored or generated code. The raw ledgers are about 320 KB of JSON from synthetic and fixture data.
• Public wording: no "beta", "coming soon" or "unreleased" wording, and no prospect or customer names in the diff.
• Approval spoofing: a teach_run approval cannot be forged by the client. _held_by_person is set only by the MCP layer from the person's confirmed binding, and a stale binding is re-checked against a fresh first_commit before anything runs.
• Comment and blank lines: comment and blank lines are filtered the same way in run(), _first_line() and first_commit(), so "line 1" means the same line in all three.

[abrichr]

AI re-review (coordinator)

Verdict: ready to merge. All six blockers from the first review are closed at 4d74a9d0, and each has a test.

I ran a direct probe of RecordReadBack and the harness helpers (no browser) at this head:

case	before (fe7df6e9)	now
1. overwrite a starting record that matches expect	landed	wrong_record (stops)
1. delete it and create a new matching one	landed	wrong_record (stops)
1. same, in a store without ids	landed	wrong_record (stops)
1. a new record next to an untouched starting one	landed	landed
2. duplicate save, keyed store	landed	duplicate (stops)
2. duplicate save, store without ids	no_change	duplicate (stops)
3. save shows nothing; the write lands after the poll window	next save allowed	unconfirmed; next save refused with landed_late, nothing sent
3. write lands inside the window (0.5 s of 1 s)	landed	landed
4. the save click raises, and a duplicate landed	not read back	read back, ReadBackRefused raised from the original error
5. mark() read fails	fell back to the start read, landed	unreadable, run stops before the save is sent
5. no starting read	every save unreadable, run went on	run 1 does not start
run-1 record later moved to another patient	n/a	wrong_record
named record lands with an empty note	landed	incomplete

Tests: tests/test_code_action.py and tests/test_compile_agent.py pass (115), tests/e2e/test_code_agent_default_e2e.py passes (5 of 5, real MockMed in a browser, run serially at host load 22 to 46). Ruff is clean on the changed files. New tests cover each blocker: test_any_change_to_a_starting_record_is_the_wrong_record, test_a_duplicate_save_stops_the_run[with_ids], test_a_save_that_lands_late_stops_the_next_save, test_a_save_that_raises_is_still_read_back, test_an_unreadable_read_or_a_missing_start_read_stops_the_run, test_run_1_does_not_start_without_a_starting_read, test_a_save_without_an_input_value_is_incomplete.

6. Public copy: CAPABILITIES.md, llms-full.txt, the teach README and the teach_run description now say "every save the gate detects", say the item 40 write landed before the stop, and state the scoreboard's silent_wrong 1 with a link to HAND_REVIEW.md. The unrelated flow: induce search-open-act-return and row-varying loops, each row checked on its own record #1435 row is gone from llms-full.txt. PROTOCOL Amendment 13 records that the arm 7 runs used the earlier rules and why their verdicts hold.

Also fixed: meta.json now records the code each run used (code_at_run, including the uncommitted recorder change in the admit run), and the runner writes repository-relative engine paths.

Non-blocking, for a follow-up

• scoreboard.json still adds 8 /Users/abrichr/... engine paths. The meta.json files were fixed by hand, but scoreboard.json was not regenerated. The next scoreboard regeneration with the new schema.engine() clears them.
• landed checks values anywhere in the record. An input value that matches some other field counts as held. Compare by field where the input maps to one.
• Deleting the run's own record in a later save reads as unconfirmed, not as a stop. No starting data is at risk, and the final record check catches it.
• The teach_run start timeout is unchanged. The new 3 s target wait helps with slow pages, but the app did not open in time under heavy load can still flake in the merge queue. It passed here.
• The other non-blocking items from the first review (final read-back when run 1 ends early, settle read after the first change, approval binding to page state, MCP approval e2e, shell_isolation in teach_start, read cost on large tables) still apply.

Action: CI's required checks are queued with no failures, so I'm enabling auto-merge (merge commit).