Skip to content

undo propose: verified undo steps for each save, approved once by a person - #1346

Merged
abrichr merged 15 commits into
mainfrom
claude/gracious-chandrasekhar-43a0c5
Oct 20, 2026
Merged

abrichr merged 15 commits into
mainfrom
claude/gracious-chandrasekhar-43a0c5

Conversation

@abrichr

@abrichr abrichr commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What this does

After a stop, openadapt-flow undo (#1263) could reverse a change only with an undo a person wrote or recorded by hand. This adds undo propose: for each saving step, Flow finds candidate undo steps, tries each one on a declared test copy, and writes only the ones that pass into the bundle, as proposals. A person approves a proposal once (undo accept). After a stop it is offered like a declared undo, and each run still needs a person's approval then (undo approve). Flow never runs an undo on its own.

Where candidates come from

  • Other recordings (--from-recording): a compiled recording of a person deleting, cancelling, voiding or removing a record. If its steps declare no effect, Flow adds the save's inverse (the record the save created is gone).
  • Captured traffic (--recording, its network.jsonl from feat(flow): API steps compiled from a recording's network capture #1269): a DELETE the recording sent to the created record, else, for a JSON POST that returned an id, DELETE <collection>/{undo_record_id}. A JSON PUT/PATCH field edit gets the same request with the old value.
  • The old value: for a field edit typed from a parameter, the same steps again with the value read back before the change.
  • App controls: a Delete/Cancel/Void/Remove control on a page loaded after the save is reported, not driven. Record a person using it.

Browser form posts and GraphQL mutations are reported with that reason; Flow builds API undos only from JSON requests.

How a candidate is verified (test copy only)

Same gate as induce --probe-on (#1276/#1289): test_copy.declared: true, a reset, a reward contract, no production profile, a fresh session per run.

  1. Reset, then read every record through the reward contract's oracle. When earlier steps also save, run the steps ahead of the save first, so "before" is the state just before this save.
  2. Run through the save with the deployment's effect verifier. It must complete and change something.
  3. Run the candidate undo in a new session, the way undo run does. Its declared effect must be confirmed by an independent read-back (judge_undo_report, the same judge undo run uses).
  4. Read every record again. Pass only if the records equal step 1: the saved record restored, and no other record changed (the collateral guard names records the save never touched).
  5. Reset.

Evidence (snapshot digests and counts, how each run ended, the reason) holds no record values. A failed candidate is reported with its reason and never written.

Finding the record at undo time

A proposed undo can carry locate (find the record the save created) or restore (old field values). Just before the save, the replayer keeps what the undo needs from the pre-action read it already makes for the save's record check, in <run-dir>/undo/prior/ (owner-only; never in the report). At undo time, the one matching record that wasn't there before is the target; zero or several refuse before anything runs, and the approval stays unused.

Measured

Per saving step, on the synthetic apps (live: real browser, network capture, app declared a test copy) and on the local fixtures' existing recordings (offline: nothing run). Details and reasons: packages/flow/benchmark/undo_proposal/.

App Run Saving step Verified undo Why not, and what would close the gap
MockMed live Save Encounter Yes: DELETE /api/encounter/{undo_record_id} (captured traffic)
MockLoan live Authorize Disbursement Yes: DELETE /api/disbursement/{undo_record_id} (captured traffic)
MockLoan, fee-row mode live Authorize Disbursement No The save also writes a fee row; the DELETE leaves it. Closes with a recording of the app's own reversal path.
OpenEMR fixture offline step_039 No No record check bound. Closes with binding one.
OpenEMR fixture offline step_040 (the save) No Browser form post; only a Cancel control after the save; no recorded delete. Closes with a recording of the app's delete path.
openIMIS fixture offline step_015 (the save) No No network capture, no recorded delete; live fixture not started. Closes with a recording of the delete/cancel path or --capture-network.

Total: 2 of 6 saving steps have a verified undo, both plain saves on the synthetic apps. Each verified run: records after the undo equal the records before the save (same snapshot digest), 0 model calls. The OpenEMR and openIMIS rows are the honest coverage number on real-app fixtures today: none, for the reasons listed.

Also found: the compiler doesn't label MockLoan's "Authorize Disbursement" click a write. Binding its record check (as an operator does) makes it one; the risk heuristic gap is a follow-up.

Changes

  • openadapt_flow/undo_proposal.py (new): candidates, test-copy verification, write_proposals, accept_proposal, DeploymentUndoTestCopy.
  • ir.py: UndoStep.locate, .restore, .proposal, source synthesized; unset fields are left out of the bytes, so existing bundle digests don't change.
  • after_stop.py: unapproved proposals are not offered and not named in the ledger; keep_prior_state, resolve_undo_params, judge_undo_report, save_undo_steps; values are resolved before the approval is claimed.
  • runtime/replayer.py: one hook after the save's pre-action read (GUI and API paths); acts only when the bundle has a locating or restoring undo for the step, never raises, never changes the result.
  • cli_undo.py: undo propose, undo accept (interactive terminal only); undo run reads records through --config's effect verifier for a locating undo; undo list names proposals waiting for approval.
  • Docs: AFTER_A_STOP.md (Proposed undo steps), LIMITS.md, the CAPABILITIES.md row and command list.
  • benchmark/undo_proposal/ (harness, README, results).

Tests

  • tests/test_undo_proposal.py (22): end to end against the real MockMed record store and a small field-edit store, with the real Replayer, REST read-back and API actuator: a network undo verified, written, not offered until accepted, then approved and run after a real stop; a wrong endpoint fails and is never written; a recorded delete that also removes an earlier record fails the collateral guard; a field edit gets a verified restore; a restore that touches another record fails; plus candidate reasons, locate refusals (0 or 2 new records, unreadable store), evidence tampering, byte-identical declared undos, CLI refusals.
  • Unchanged suites run locally one file at a time: test_after_stop_mockmed (22), test_replayer (119), test_replayer_api_actuator (25), test_replayer_effects (13), test_api_tier_web_form (31), test_probe_induction (21), test_cli_new_commands (36), test_bundle_schema_v2 (41), test_policy (33), test_capabilities_map. The full suite runs in CI.

Synthetic data only.

🤖 Generated with Claude Code

abrichr and others added 11 commits October 19, 2026 11:06
undo propose finds candidate undo steps for each saving step (another
recording, captured traffic, the field's old value), tries each on a
declared test copy, and writes only the ones that restore every record
as proposals. undo accept is a person's one-time approval; each run
still needs approval after a stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hint

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…penIMIS rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr marked this pull request as ready for review October 19, 2026 17:26
abrichr and others added 4 commits October 19, 2026 15:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rasekhar-43a0c5

# Conflicts:
#	llms-full.txt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	CAPABILITIES.md
#	llms-full.txt
#	packages/flow/public-artifacts.json
@abrichr

abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

Included in batch #1384, which runs CI once for several reviewed PRs. This PR is marked merged when #1384 merges. Please don't push here unless asked; a new commit leaves it out of the batch.

@abrichr
abrichr merged commit b7a25ea into main Oct 20, 2026
30 of 40 checks passed
@abrichr
abrichr deleted the claude/gracious-chandrasekhar-43a0c5 branch October 20, 2026 02:18
abrichr added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant