[abrichr]

Adds a pre-registered leaderboard of computer-use agents on the 37 public Medplum cases, run in Medplum's own web app.

What is in it (packages/flow/benchmark/fhir_medplum/leaderboard/):

• PROTOCOL.md: the protocol, frozen before any scored run. Its SHA-256, and the hashes of every harness file, are committed to openadapt-internal (evidence/write-leaderboard-2026-11/PREREGISTRATION.sha256).
• browser.py, run.py: the harness signs in to the Medplum web app, routes the app's FHIR calls through the public fault proxy inside the browser, and scores each run with public/oracle.py. Anyone can run their own agent with --agent py:<module>:<function>.
• agents/: adapters for Claude computer use, OpenAI's computer tool, Gemini's computer use (built, not run: no API credit) and Fara-7B on MLX. The Fara prompt is vendored MIT text from microsoft/fara.
• score.py: the three metrics (said done with the record wrong, collateral writes said done, pass^k with k=3), bootstrap intervals over cases.
• REPORT.md, DEVIATIONS.md, results/: every scored row with its transcript and first and last screenshots, the excluded signed-out runs, and the spend ledger.

Results (3 runs of each case):

Agent	Said done, record wrong	Collateral writes said done	pass^3
claude-sonnet-5-5	25.2% (CI 13.5–37.8)	20 of 20	56.8%
gpt-6.1-sol	29.7% (CI 16.2–46.0)	30 of 30	62.2%

The intervals overlap; the data do not rank the two. Neither misreported a clean case. Fara-7B's rows are still running locally and will be added by a follow-up commit.

Three deviations are logged with reasons: a host-load guard, Fara run one process at a time, and 188 runs rerun after the app's one-hour session expired under them.

Spend at list prices: $64.78 (claude) and $20.48 (openai), including the excluded runs.

[abrichr]

Correction (841d1e7, deviation 4): the verifier's read token expired an hour into each process, so the live shadow column judged only 59 of 222 runs. The shadow column now uses verdicts replayed from Medplum's stored history (#1795): claude flagged 25 of 28 misreports with 26 of 72 false flags, and openai 30 of 33 with 11 of 68. The end-state check uses a separately renewed token, so the three ranked metrics are unchanged. public/run.py now renews the verifier's token before each baseline and each check.

[abrichr]

From: Grok, To: coordinator

APPROVE at 99e3860. Checked in a detached worktree with main merged in locally (clean merge, not pushed).

Checked:

• Scores reproduce. python -m benchmark.fhir_medplum.leaderboard.score regenerates LEADERBOARD.json and LEADERBOARD.md byte for byte, with no diff. Claude: 25% said done with the record wrong (CI 14–38), pass^3 57%. OpenAI: 30% (CI 16–46), pass^3 62%. Both had 111 runs.
• Packaging. results/ is excluded from both the wheel and the sdist, and it is listed in public-artifacts and in the approved directories of check_release_consistency.py. The script passes. test_release_contract, test_release_publication, test_bench_medplum_public and test_release_ci_gate: 119 passed. CI is green.
• Privacy scan of the full diff (1,912 files, 44 MB of text):
  • no API keys, JWTs, bearer tokens or AWS account ids;
  • only example.org synthetic emails;
  • no private IPs.
  • The sign-in credentials come from the fixture's generated state and never reach results/.

P2:

• The pre-registration can't be checked from outside. The protocol hash and harness hashes are only in a private repo, so outside readers can't verify the "frozen before any scored run" claim. Publishing the hash file itself (hashes only) next to PROTOCOL.md would fix that.
• The PR body is out of date on Fara. It says Fara's rows are still running, but results/fara already has 57 runs (the first half), and LEADERBOARD.md shows them: 26% said done wrong, pass^3 0%. The body and REPORT should say this row covers half the cases.

P3:

• Ruff errors in leaderboard/file_times.py (E401, I001, E702). CI doesn't lint this path.
• Local paths in the committed tracebacks. Two tracebacks in results/ include a local home-directory worktree path. That's harmless, but they could be trimmed.
• CAPABILITIES says "On main" before merge. That's the usual pattern, but it's only true once merged.
• No unit tests for the new code. score.py and common.py have none; the reproduced board is the only check.