Skip to content

verify-run: frame conditions (nothing outside the declared delta changed) - #1443

Merged
abrichr merged 3 commits into
mainfrom
verification-coverage/frame
Oct 20, 2026
Merged

abrichr merged 3 commits into
mainfrom
verification-coverage/frame

Conversation

@abrichr

@abrichr abrichr commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Part 1 of the design in #1434 (docs/VERIFICATION_COVERAGE.md): frame conditions for verify-run and the guarded save.

Problem

In #1365's OpenEMR fault run, verify-run reported verified on all 5 saves where another chart was edited in place. #1375 added forbid_collateral_change, but it judges one effect at a time. Two effects on one read set flag each other's targets, a stray added record passes, and it can't tell a legitimate edit by another user from a stray write.

Change

  • runtime/effects/frame.py: judge_frame compares the whole read set before and after the save, once per contract. Any record that no effect declares and that was added, removed or changed is a frame change. So is a pre-existing target changed outside the fields the effects name.
  • Attribution (optional frame.attribution):
    • audit_read: a second read-only read of the app's audit log. Entries new since the baseline are the window.
    • record_field: a modified-by field plus a version or ETag on each record.
    • A change attributed to another user is a concurrent edit and doesn't count. A change by the bot's own ids (self, literals or params), or one nothing explains, is collateral. With audit_read, a bot log entry on an undeclared record is collateral even if the read shows no change.
  • verify-run:
    • The audit read is stored with the baseline.
    • With a baseline, the frame judge replaces the per-effect collateral rules, unless the file sets those rules explicitly. A frame refutation also makes effect_verdict REFUTED, so older readers see what they saw before.
    • Without a baseline the frame is not_checked and the run is not verified.
  • Receipt: a new optional signed field, checked: {delta, frame, source}. source is instruction here (the --params); provenance comes in a later PR. verified requires every part, which receipt_problem and countersign verify both enforce. evidence.frame holds counts and field names only, never records or actors.
  • countersign verify prints a "Checked" line. qualify ignores frame:, so one verify.yaml still serves both commands.

Safety review: changes to when verified is issued

Coordinator review requested (local_bcd23d2d) before merge.

  1. No baseline means no verified. Before, an absolute-count contract could verify without one.

  2. A record added to the read set that no effect declares now refutes.

  3. A wrong-patient write is reconciliation_required, no longer halted_before_effect. A record was written; test_wrong_patient_is_refuted changed accordingly.

  4. Two effects on one read set no longer refute each other's targets.

  5. The only loosening, with the review's three conditions (commit 81034b7, a test each):

    • (a) evidence the bot can't write only: the app's audit log, or record fields declared server-set per app (server_set). A field the run writes, or a run parameter names, refuses attribution.
    • (b) the executor's own id is one of the bot's ids, never "another user".
    • (c) another actor's change to a record the save declares blocks verified (target_changed_by_other); only changes outside the declared delta are excused.

    Before the conditions: a change that an audit log or a modified-by field with a new version attributes to a named non-bot user is not collateral. Unattributed changes, and an unreadable log, still refute.

Flow replay's verified is unchanged by this PR; the design doc says so and plans it separately.

Tests

  • tests/test_frame_conditions.py (28): the judge, both attribution kinds, verify-run end to end on MockMed, and a SQLite store with its own audit log where another user's edit stays verified and the bot's own edit is refuted.
  • Ran locally: 81 receipt, seal, countersign, qualify, guard, verify-run, effect and record-check test files. 1756 passed, 1 failed. The failure (test_qualification_gate_campaign::test_cli_durable_authority_failure_preserves_the_original_failure) fails the same way on a clean origin/main checkout here.
  • ruff, and mypy on the changed modules.

The fresh OpenEMR and openIMIS benchmark runs follow in their own PR.

🤖 Generated with Claude Code

abrichr and others added 2 commits October 20, 2026 13:07
…anged

A record check said a save was right while another chart was edited in
place (#1365). The frame judge compares the whole read set before and
after the save, once per contract: any record outside the declared
records that was added, removed or changed, or a target changed outside
its declared fields, is collateral. A change that the app's audit log or
a modified-by field with a new version attributes to another user is a
concurrent edit and does not count; a change by the bot's own ids or one
nothing explains does.

- runtime/effects/frame.py: FrameSpec, FrameAttribution (audit_read,
  record_field), judge_frame.
- verify-run: optional frame: section; the audit read is taken with the
  baseline; the frame judge replaces the per-effect collateral rules when
  a baseline exists; without one the frame is not_checked and the run is
  not verified.
- Receipt: optional signed field checked {delta, frame, source}; source
  is 'instruction' (the --params). verified requires every part.
  evidence.frame carries counts and field names only.
- countersign verify shows and checks it; qualify ignores frame:.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaffold-verifier reads each table by id and by content
(records_drop_fields: [id]). The frame judged the twin as its own slice,
so the saved row looked like an undeclared addition there. Surfaces that
differ only in dropped fields now share their declared records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The safety review of #1443 set three conditions on excusing a change as
another user's concurrent edit:

- record_field attribution must name fields the app sets on the server,
  declared per app in server_set; a field the run writes, or a run
  parameter names, refuses the attribution (every change unattributed).
- the executor's own id joins the bot's ids: never another user.
- another actor's change to a record the save declares blocks verified
  (the saved value may not stand); only changes outside the declared delta
  are excused.

Server-set bookkeeping fields are ignored as content: they say who wrote,
not what changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

Coordinator safety review: approve at 81034b7.

  • Attribution comes only from per-app server-set fields or the audit log. A field the run writes, or a run parameter names, refuses attribution.
  • The executor's id counts as the bot's own.
  • Another actor's change to a declared record blocks VERIFIED (reconciliation_required, target_changed_by_other).
  • Unattributed changes still refute.
    Merge when green.

🤖 Generated with Claude Code

@abrichr
abrichr enabled auto-merge October 20, 2026 18:10
@abrichr
abrichr added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 3b294c2 Oct 20, 2026
38 of 39 checks passed
@abrichr
abrichr deleted the verification-coverage/frame branch October 20, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant