Write leaderboard: computer-use agents in Medplum's web app, pre-registered

CAPABILITIES.md

@@ -162,6 +162,7 @@ improvising or retrying when something is off.
 | `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) |
 | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
 | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
+| A pre-registered leaderboard of computer-use agents doing the same 37 public cases in Medplum's own web app: the app's FHIR calls pass through the fault proxy inside the browser, the harness signs in, and the end-state oracle scores how often an agent says done when the record is wrong (`python -m benchmark.fhir_medplum.leaderboard.run --agent <claude|openai|fara|py:...>`) | On main ([#1737](https://github.com/OpenAdaptAI/OpenAdapt/pull/1737)) | Real app (Medplum 5.2.8 web app, 3 runs per case: said done with the record wrong in 25.2% of runs for Claude Sonnet 5.5 and 29.7% for GPT-6.1-sol, intervals overlapping, and 18.2% for local Fara-7B, which reached the 40-minute limit in 84 of 110 runs; neither API agent misreported a clean case; every collateral write was said done; protocol and hashes committed before any scored run) | [README](packages/flow/benchmark/fhir_medplum/leaderboard/README.md), [REPORT.md](packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md), [PROTOCOL.md](packages/flow/benchmark/fhir_medplum/leaderboard/PROTOCOL.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
 | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |

llms-full.txt

@@ -565,6 +565,7 @@ improvising or retrying when something is off.
 | `void` in an ensure-record contract: values that make a record assert nothing (`status="entered-in-error"`). A record holding one is never the person, is not a duplicate, and a created one needs review; it names values to leave out, never values to keep, and can't name an identity field | On main ([#1657](https://github.com/OpenAdaptAI/OpenAdapt/pull/1657)) | Real app (Medplum 5.2.8: 8 live cases written in advance, 8/8 with the line, 5/8 and one silent pass without it; frozen blind Immunization suite 24/24); adversarial review recorded | `openadapt_flow/ensure_record.py`, [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/void/results/RESULTS.md), [REVIEW_VOID.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/void/REVIEW_VOID.md) |
 | Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
 | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 4 false stops, all corrections that changed the record's identity, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
+| A pre-registered leaderboard of computer-use agents doing the same 37 public cases in Medplum's own web app: the app's FHIR calls pass through the fault proxy inside the browser, the harness signs in, and the end-state oracle scores how often an agent says done when the record is wrong (`python -m benchmark.fhir_medplum.leaderboard.run --agent <claude|openai|fara|py:...>`) | On main ([#1737](https://github.com/OpenAdaptAI/OpenAdapt/pull/1737)) | Real app (Medplum 5.2.8 web app, 3 runs per case: said done with the record wrong in 25.2% of runs for Claude Sonnet 5.5 and 29.7% for GPT-6.1-sol, intervals overlapping, and 18.2% for local Fara-7B, which reached the 40-minute limit in 84 of 110 runs; neither API agent misreported a clean case; every collateral write was said done; protocol and hashes committed before any scored run) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/README.md), [REPORT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md), [PROTOCOL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/PROTOCOL.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
 | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |

packages/flow/benchmark/fhir_medplum/leaderboard/DEVIATIONS.md

@@ -0,0 +1,97 @@
+# Deviations from the pre-registered protocol
+
+Every change made after the protocol was frozen (openadapt-internal
+3f6fe69a, harness 7a320e98), with its reason and its effect on the scores.
+
+## 1. Host-load guard (2026-11-02, about 22:00 EST)
+
+**What changed:** before each run, `run.py` now waits while the host's
+1-minute load average is over 35, and records the load at the start and end
+of the run (`host_load` in each row). The machine's operating rule is to
+keep load under 35 and to pause timed work above 40. The host load reached
+about 120, mostly from other jobs on the same machine.
+
+**Effect on scoring:** none on the definitions. The metrics, claims, oracle
+and limits are unchanged. The guard can only make a run wait to start; it
+never changes a run once it has started.
+
+**Runs affected:** the 14 rows written before the guard (claude r1 P01-P06,
+openai r1 P01-P08) ran while the load was high and have no `host_load`.
+The protocol has no rule to rerun on load, so they are kept as they are and
+flagged in the report. A slow host can only make a run more likely to reach
+the 2,400-second limit, which ends it with no claim. The report states
+whether any of these 14 rows did.
+
+**Runs in flight when the processes were restarted** were not written and
+started again from the beginning, the same as the protocol's rule for a
+harness failure. No outcome of those runs was seen before they restarted.
+
+## 2. Fara runs one process at a time (2026-11-02, about 22:00 EST)
+
+**What changed:** the protocol split Fara's cases across two processes
+(P01-A06 and A07-AP12) on two stacks. Because the machine's load was high,
+they now run one after the other: P01-A06 first, then A07-AP12, each in
+the protocol's order.
+
+**Effect on scoring:** none. Only the wall-clock time changes.
+
+The second process ran on the first process's stack (port 9820) instead of
+its own (9830), because the host's free disk was under the lane's 30 GiB
+gate for starting a stack. Each run empties and seeds the project first,
+so the stack carries nothing from one run to the next. Before the second
+process started, the stack's history was dumped for deviation 4's replay
+(`results/fara/frozen-9820.jsonl.gz`).
+
+## 3. Signed-out runs rerun; the session is renewed (2026-11-03, about 00:00 EST)
+
+**What happened:** the harness signed the agent's user in once per process.
+Medplum's session lasts an hour, and nothing renewed it, so every run that
+finished more than an hour after its process signed in had been signed out:
+the app showed its sign-in page and the agent, correctly, said NOT DONE.
+That is the harness's failure, not the agent's.
+
+**What changed:** before each run the harness signs in again if its
+session is more than 15 minutes old, so a run (at most 40 minutes) always
+ends inside the session. A run that starts or ends on the sign-in page now
+raises a harness error, which the pre-registered rule retries up to twice.
+`score.py` now also leaves out a run the harness failed three times
+(`infra_note`), as PROTOCOL.md already said; before, it scored such a run
+as no claim. No run had one.
+
+**Runs affected:** every row written more than 3,600 seconds after its
+process's last sign-in (the time in its `meta-*.json`) is excluded and run
+again: claude 96 rows (r1 A03 onwards, all of r2 and r3), openai 84 (r1
+AP03 onwards, all of r2 and r3), fara 8 (r1 P04 onwards). This rule picks
+exactly the rows whose end page was the sign-in page, plus fara r1 P04,
+which ran across the expiry. The excluded rows and their evidence are kept
+in `results/<agent>/excluded-signed-out/` and listed in
+`results/excluded-signed-out.json`; none is scored. Their spend stays in
+the spend ledger and counts against each agent's share.
+
+## 4. The verifier's token is renewed; the shadow column is replayed (2026-11-03, about 20:00 EST)
+
+**What happened:** the harness set the verifier's read token
+(`BENCH_RECORD_TOKEN`) once per process, and Medplum's tokens last an
+hour. Deviation 3 renewed the browser's session but not this token. From
+about an hour into each process the verifier's reads were refused (HTTP
+401). Failing closed, it reported `needs_review` on 92 of 111 claude rows
+and 71 of 111 openai rows without judging them, and those rows counted as
+flagged in the shadow column.
+
+**Not affected:** the end-state check and the read-back. They read with the
+harness's admin token, which is renewed every 10 minutes. No row's
+end-state or read-back read was refused, so the three pre-registered
+metrics are unchanged.
+
+**What changed:** `public/run.py` renews the verifier's token before its
+baseline and before its check. For the 222 claude and openai runs, the
+shadow column now uses the verifier's verdicts replayed from Medplum's
+stored history of the same runs, with the same contracts (session
+verifier-specificity, `../leaderboard_rescore/`, column "main", per-row
+in `results/shadow-replay.json`). The replay reproduces all 59 live
+verdicts the verifier did judge. `score.py` uses a replayed verdict where
+one exists and keeps the live column (`shadow_live`) beside it.
+
+**Fara:** its first process (P01-A06) ran before the fix, and its shadow
+column is replayed the same way. Its second process (A07-AP12) runs with
+the fix.