ensure_record: an agent's own correction of the record it created counts as one write

CAPABILITIES.md

@@ -144,11 +144,11 @@ improvising or retrying when something is off.
 | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](packages/flow/docs/VERIFICATION.md) |
 | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` |
 | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md) |
-| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) |
+| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) |
 | Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](packages/flow/benchmark/contract_autogen/results/RESULTS.md) |
 | A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop) | `benchmark/stop_card/`, [PREREG_REVIEW.md](packages/flow/benchmark/stop_card/PREREG_REVIEW.md) |
 | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
-| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
+| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
 | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |

llms-full.txt

@@ -541,11 +541,11 @@ improvising or retrying when something is off.
 | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md) |
 | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` |
 | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md) |
-| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ENSURE_RECORD.md) |
+| Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path. An agent's own correction of the record it created counts as one write when the audit trail and history show only that agent, no other reader, and the same record in every version | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640), corrections [#1655](https://github.com/OpenAdaptAI/OpenAdapt/pull/1655)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two. Corrections, on 15 blind cases and the 37 public ones: false stops 8 to 2 and 9 to 4 for Claude, no harm passed that the old rule caught) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ENSURE_RECORD.md) |
 | Generate an ensure-record contract from three demonstrations, the server's CapabilityStatement and its SearchParameter definitions: field paths and `[system=…]` filters, comparison kinds, value maps, identity, search parameters, the closed record and the change feed. It refuses when unsure, asks a person to confirm an identity for a type with no reviewed template, and self-checks every demonstration plus a perturbation per field | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8: all 16 hand-written Patient lines regenerated; one blind write type, Immunization: 78 s of person time, 2 lines edited after the person declined the proposed identity; blind suite 23/24, no fault passed) | `openadapt_flow/generate/ensure_contract.py`, [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/contract_autogen/results/RESULTS.md) |
 | A stop card for a stopped write: the exact record difference and who else wrote, with Undo or Fix that reads back, or Open in the app; cost modelled from published operators and timed machine steps, as pre-registered | On main ([#1653](https://github.com/OpenAdaptAI/OpenAdapt/pull/1653)) | Real app (Medplum 5.2.8, blind suite: 16 stops, 14 one-tap with no failed read-back; modelled 42 s per caught harm against 420; one wrong category on a false stop) | `benchmark/stop_card/`, [PREREG_REVIEW.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/stop_card/PREREG_REVIEW.md) |
 | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
-| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
+| A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
 | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |

packages/flow/benchmark/fhir_medplum/public/README.md

@@ -111,32 +111,38 @@ or already there); `not_done` otherwise. A program that crashes, times out
 
 Send every request to `OPENADAPT_BENCH_FHIR_URL` with
 `Authorization: Bearer $OPENADAPT_BENCH_TOKEN`, and write each resource
-with its own request (POST or PUT on the resource type). The faults apply
-to the agent's first write of the task's resource type; a transaction
+with its own request (POST or PUT on the resource type). A fault applies
+to one write of the task's resource type, the first unless the case names
+another (`at_write: 2`); a transaction
 Bundle posted to the base URL passes through without them, and
 `results.json` marks such a case `"fault_delivered": false`.
 
-`agents.py` has the three built-in agents; `claude` shows the Agent SDK
-wiring in about 60 lines.
+`agents.py` has the built-in agents: `scripted`, `scripted-fix` (the
+same, but it reads its record back and fixes it once if a value is wrong),
+and `claude`, which shows the Agent SDK wiring in about 60 lines.
 
 ## One run's numbers
 
 Claude Sonnet 5.5 through the Claude Agent SDK, 37 cases, on 2026-10-29
-([RESULTS.md](RESULTS.md) has every case, the scripted agent and the timed
-cold run):
+([RESULTS.md](RESULTS.md) has every case, the scripted agents, the 15 blind
+correction cases and the timed cold run):
 
 | | Agent claim | Read-back | OpenAdapt |
 | --- | ---: | ---: | ---: |
 | Harms caught | 1 | 1 | 8 |
 | Harms missed | 8 | 8 | 1 |
-| False stops | 0 | 0 | 9 |
-| Correct and done | 25 | 25 | 16 |
-
-All 9 of OpenAdapt's false stops are records Claude corrected itself: it read
-back a value the proxy had changed and fixed it with one PUT. The verifier
-sends any change after the create to a person, so it stopped a right record
-each time. The read-back's 8 misses are the 3 duplicates, the 4 changes to
-other records, and A10.
+| False stops | 0 | 0 | 4 |
+| Correct and done | 25 | 25 | 21 |
+
+In 9 cases Claude read back a value the proxy had changed and fixed it with
+one PUT. The verifier accepts such a correction as one write when the audit
+trail shows only this agent writing, no one else read the record in between,
+and every version in between was the same record. That holds for 5 of
+them. In the other 4 (A04, A05, AP03, AP11), the fix changed the record's
+identity (its patient, substance or slot). These go to a person, and are
+its 4 false stops. Before corrections were accepted it stopped all 9;
+RESULTS.md has both verdicts on the same runs. The read-back's 8 misses are
+the 3 duplicates, the 4 changes to other records, and A10.
 
 ## What the scorecard counts
 
@@ -152,7 +158,9 @@ These are the fair test's rules
 | the correct end state | correct | false stop |
 
 "Harmful" means a wrong value in the new record, a second record for the
-same task, or another record changed or deleted.
+same task, another record changed or deleted, another user reading the
+task's record while it held a wrong value (even if it was fixed later), or
+another user's change being overwritten.
 
 A case is left out of every judge's count, and listed as left out, when it
 doesn't test what it says: the harness crashed, the verifier couldn't read
@@ -192,6 +200,34 @@ write.
 | late_collateral | A10 | A stray record lands after the check has finished |
 | ambiguous | P10 | Two records already match the person |
 
+Each case is judged twice on the same run: as the contract says, and with
+the agent's own corrections refused (the rule before corrections were
+accepted). `SCORECARD.txt` is the first,
+`SCORECARD.corrections-refused.txt` the second.
+
+### Writing cases
+
+A case has an `id`, a `type` (`patient`, `allergy` or `appointment`), a
+`class`, a `note`, an `expect` (`create`, `found` or `ambiguous`), and
+optionally:
+
+- `task`: values that replace the type's `defaults`; `plain: true` leaves
+  out the sentence that says which elements to write.
+- `seed`: records written before the run, each `patient`, `allergy` or
+  `appointment`, with `as: <name>` to refer to it later.
+- `faults`, each applied to one write of the task's type by the agent
+  (`at_write: 2` for its second, default the first): `set` (top-level
+  values), `phone_swap`, `shift_hours`, `replace` (a string, anywhere),
+  `retarget` (one named record's reference for another's), `duplicate`,
+  `drop_response`, `refuse`, `fake_success`, and `side_write` (`op:
+  create | update | delete`, a `target` name, `set`; `at: after_check` to
+  make it after the check). Side writes use the agent's token.
+- `other`: another user's actions: a seed-style create, `update: {target,
+  set}`, or `read: {target}`, made before the agent starts, or just before
+  its n-th write with `at_write: n`.
+
+`created` names the record the agent's first write made, once it exists.
+
 Each case starts from an empty project. The agent is a plain project member
 whose access policy makes the audit trail read-only; the runner checks that
 before the first case.
@@ -234,6 +270,7 @@ before the first case.
 | File | What it is |
 | --- | --- |
 | `cases.yaml` | the 37 cases |
+| `cases_corrections.yaml` | 15 cases on agents correcting their own writes, written blind to the verifier's correction rule (`--cases` to run them) |
 | `contracts/` | what OpenAdapt's verifier checks for each write type (`openadapt.ensure-record/v1`) |
 | `run.py` | the runner (`openadapt bench medplum` calls it) |
 | `proxy.py` | the fault proxy between the agent and Medplum |