From 2dc472a48172251bb6e369a8074a4d449e4e9bbd Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 09:52:45 -0400 Subject: [PATCH 1/2] fix(flow): halt a relabelled Save on the Save, not the scroll before it (#1331) A closed-loop scroll stops when the next step's anchor resolves. When the vendor relabels that control, the scroll used up its budget and the run halted on the scroll, so self-repair refused it ("not a click"). Now, when every readiness probe missed because no direct deterministic rung found the target, a non-consequential scroll hands the miss to the next click. That click accepts only direct rungs (structural, template, OCR; no model) and halts with the ordinary "Could not resolve target" error and no input. A target that resolved but was out of place or failed its identity check still halts on the scroll, as does a consequential scroll or one in a program graph. The RECONCILIATION_REQUIRED outcome on openIMIS is correct: the Tab key and the unlabelled picker click carry risk_review_required, so a write cannot be ruled out. Self-repair's refusal now names the unsettled steps and says a person can review them with `qualify set-risk`; once reviewed, the same halt is HALTED_BEFORE_EFFECT and repairable (tested). The reliability classifier files the handed-off halt as resolution_failed. Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 1 + llms-full.txt | 8 +- packages/flow/docs/SELF_REPAIR.md | 13 ++ .../openadapt_flow/benchmark/reliability.py | 7 +- .../flow/openadapt_flow/repair/self_repair.py | 28 ++- .../flow/openadapt_flow/runtime/replayer.py | 127 ++++++++++-- .../flow/tests/test_relabelled_save_halt.py | 191 ++++++++++++++++++ packages/flow/tests/test_replayer.py | 66 +++++- 8 files changed, 409 insertions(+), 32 deletions(-) create mode 100644 packages/flow/tests/test_relabelled_save_halt.py diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 8259b43fe3..93ab08cd60 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -189,6 +189,7 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1331](https://github.com/OpenAdaptAI/OpenAdapt/issues/1331)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | | Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | diff --git a/llms-full.txt b/llms-full.txt index 623dfd5b6b..3bbedf8001 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -375,7 +375,8 @@ shipping in the next release; a pull request number means it is being merged. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles the verified run into a program (`openadapt_flow.jit`, released; - prompt-to-program `compile-agent`, landing). Agents can drive the recording + prompt-to-program `compile-agent`, landing; the agent can act through code + instead of screenshots, [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)). Agents can drive the recording on web, native Windows, RDP and Citrix. 2. **Compiles to the fastest layer the app allows.** Page structure first (DOM, Windows UI Automation, macOS Accessibility), direct API calls where the app's @@ -436,6 +437,7 @@ shipping in the next release; a pull request number means it is being merged. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | @@ -543,6 +545,7 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1331](https://github.com/OpenAdaptAI/OpenAdapt/issues/1331)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | | Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | @@ -553,6 +556,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | | Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Synthetic | `compile-agent` | @@ -574,7 +578,7 @@ Each `openadapt-flow` command, grouped by the capability it serves. A test fails when a command is added without being listed here. - **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. +- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. - **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). - **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). diff --git a/packages/flow/docs/SELF_REPAIR.md b/packages/flow/docs/SELF_REPAIR.md index aa7fd51380..e67022ee4e 100644 --- a/packages/flow/docs/SELF_REPAIR.md +++ b/packages/flow/docs/SELF_REPAIR.md @@ -27,6 +27,19 @@ Module: `openadapt_flow.repair.self_repair`. CLI: `openadapt-flow run A halt on the record check (an identity mismatch) is never repaired, and neither is an item a person declined at the approval gate. + + When a closed-loop scroll runs out of budget because the next click's + target never resolved, the halt is on that click, not on the scroll. The + click accepts only direct rungs (page structure, template, OCR) and no + model, so it sends no input. A target that resolved but was out of place or + failed its record check still halts on the scroll. + + A run that ends `RECONCILIATION_REQUIRED` is never repaired. A key press or + click whose risk nobody reviewed (`risk_review_required`) may have written, + so the run ends that way even when the halt itself sent nothing. The + refusal names those steps. A person who reviews them with + `openadapt-flow qualify set-risk` makes the same halt + `HALTED_BEFORE_EFFECT`, and self-repair can take it. 2. **Propose.** The repair agent is shown the step's intent, the recorded evidence (the target crop, its label, its place) and the live screen. It proposes one action in its own dialect. Its model calls and their cost go diff --git a/packages/flow/openadapt_flow/benchmark/reliability.py b/packages/flow/openadapt_flow/benchmark/reliability.py index 605788d3da..8deddb4e90 100644 --- a/packages/flow/openadapt_flow/benchmark/reliability.py +++ b/packages/flow/openadapt_flow/benchmark/reliability.py @@ -289,7 +289,12 @@ def classify_failure(report: Optional[RunReport], error: Optional[str]) -> dict: return {"category": "none", "root_cause_hint": None} err = (failing.error or "").lower() - if failing.resolution is None and "all resolution rungs failed" in err: + if failing.resolution is None and ( + "all resolution rungs failed" in err + # A closed-loop scroll that never found the target hands the halt to + # that target's step (#1331). + or err.startswith("could not resolve target for step") + ): category = "resolution_failed" elif "identity check failed" in err: category = "identity_mismatch" diff --git a/packages/flow/openadapt_flow/repair/self_repair.py b/packages/flow/openadapt_flow/repair/self_repair.py index 41e6485dc5..41507f4ef0 100644 --- a/packages/flow/openadapt_flow/repair/self_repair.py +++ b/packages/flow/openadapt_flow/repair/self_repair.py @@ -638,8 +638,7 @@ def run(self, bundle_dir: Path | str, run_dir: Path | str) -> SelfRepairResult: if report.transaction_outcome not in (None, "HALTED_BEFORE_EFFECT"): return self._finish( "not_repairable", - f"the run ended {report.transaction_outcome}; self-repair " - "handles only a halt before any effect", + _not_before_effect(report), evidence, report, run_dir, @@ -1760,6 +1759,31 @@ def _recorded_crop(workflow: Workflow, bundle_dir: Path, step: Step) -> Optional return path.read_bytes() if path.is_file() else None +def _not_before_effect(report: RunReport) -> str: + """Why a halt that may have left an effect is not repaired, and by whom.""" + + from openadapt_flow.transaction import steps_requiring_reconciliation + + reason = ( + f"the run ended {report.transaction_outcome}; self-repair handles only " + "a halt before any effect" + ) + unsettled = steps_requiring_reconciliation(report) + if not unsettled: + return reason + reason += f" (effect not settled at {', '.join(unsettled[:8])})" + unreviewed = {r.step_id for r in report.results if r.risk_review_required} + if unreviewed.issuperset(unsettled): + # Only steps nobody classified keep this run from an absence claim. + # A person reviews them once; the rule itself stays fail-closed. + reason += ( + "; each acted with a risk nobody reviewed, so a write cannot be " + "ruled out. A person can classify them with `openadapt-flow " + "qualify set-risk`" + ) + return reason[:1024] + + def _repairable(report: RunReport, workflow: Workflow) -> "tuple[Step, int, Any] | str": """The halted step when self-repair may handle this halt, else why not.""" diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index f50df33770..b99c3826da 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -538,6 +538,16 @@ class EgressNotPermitted(RuntimeError): #: scroll counts it as ready. SCROLL_POSITION_SLACK_FRACTION = 0.25 +#: Rungs that show the target itself on screen: what closed-loop scroll +#: readiness accepts, and all a step accepts after its scroll missed it. +_DIRECT_TARGET_RUNGS = frozenset({"structural", "template", "template_global", "ocr"}) + +#: Steps that own the halt when the closed-loop scroll before them never found +#: their target (see ``Replayer._act_scroll``). +_SCROLL_HANDOFF_ACTIONS = frozenset( + {ActionKind.CLICK, ActionKind.DOUBLE_CLICK, ActionKind.RIGHT_CLICK} +) + def _scroll_target_positioned( recorded: tuple[int, int], @@ -987,6 +997,11 @@ def __init__( self.web_form_actuator = web_form_actuator self._web_form_span_active = False self._web_form_fallback_reason: Optional[str] = None + # (step id, reason) set when a closed-loop scroll used up its budget + # because the next step's target never resolved. That step then + # resolves on direct deterministic rungs only, and halts on its own + # target, so the halt names the control that changed, not the scroll. + self._scroll_missed_target: Optional[tuple[str, str]] = None # Durable tiered runtime (RFC ยง5, Tier 3), OFF by default. When True, # run() writes a RunCheckpoint after each VERIFIED step and, on a halt, # a PendingEscalation instead of just dying -- so the run can be @@ -1531,6 +1546,7 @@ def run( else uuid.uuid4().hex ) self._field_reads = {} + self._scroll_missed_target = None self._idempotency_reservation_owned = False self._qualification_fault_mutations = [] self._qualification_fault_public_key = None @@ -6634,6 +6650,13 @@ def _run_step( new page halts. """ + # A closed-loop scroll that never found this step's target hands the + # miss to this step (see ``_act_scroll``); only this step consumes it. + target_missed: Optional[str] = None + missed = self._scroll_missed_target + if missed is not None and missed[0] == step.id: + self._scroll_missed_target = None + target_missed = missed[1] router = self._surface_router if router is None: if step.action is ActionKind.SWITCH_SURFACE: @@ -6651,6 +6674,7 @@ def _run_step( run_dir=run_dir, new_crops=new_crops, graph_ctx=graph_ctx, + target_missed=target_missed, ) refusal = router.entry_refusal(step) if refusal is None and step.action is not ActionKind.SWITCH_SURFACE: @@ -6683,6 +6707,7 @@ def _run_step( run_dir=run_dir, new_crops=new_crops, graph_ctx=graph_ctx, + target_missed=target_missed, ) # Receipt: the surface this step ran on. The entry check above proved # it is the step's declared surface; the outcome classifier re-checks @@ -6797,6 +6822,7 @@ def _run_step_on_active_surface( run_dir: Path, new_crops: dict[str, bytes], graph_ctx: Optional["_GraphStepContext"] = None, + target_missed: Optional[str] = None, ) -> StepResult: """Execute a single step; never raises (failures land in the result). @@ -7114,8 +7140,8 @@ def _run_step_on_active_surface( ) result.elapsed_ms = (time.monotonic() - t0) * 1000.0 return result - resolution, matched_region, error = self._resolve_step( - step, before_png, bundle_dir, workflow + resolution, matched_region, error = self._resolve_step_target( + step, before_png, bundle_dir, workflow, target_missed ) # Retry ladder failures with fresh settled frames until # ``step.timeout_s``: a remote app can present a settled-looking @@ -7135,8 +7161,8 @@ def _run_step_on_active_surface( run_dir, evidence_step_id, "before", before_png ) last_frame = before_png - resolution, matched_region, error = self._resolve_step( - step, before_png, bundle_dir, workflow + resolution, matched_region, error = self._resolve_step_target( + step, before_png, bundle_dir, workflow, target_missed ) if resolution is not None and matched_region is not None: assert step.anchor is not None @@ -10387,6 +10413,39 @@ def _revalidate_consequential_actuation( self._active_delivery_region = fresh_region return fresh_resolution, fresh_region, fresh_png, error + def _resolve_step_target( + self, + step: Step, + screen_png: bytes, + bundle_dir: Path, + workflow: Workflow, + target_missed: Optional[str], + ) -> tuple[Optional[Resolution], Optional[Region], Optional[str]]: + """Resolve the step's target; stricter after a scroll that missed it. + + ``target_missed`` is set when the preceding closed-loop scroll used + up its budget without this target resolving. The step then accepts + only the rungs scroll readiness accepts (direct deterministic + evidence, no model): a weaker rung would act where the scroll could + not find the target. Otherwise the step halts on its own target with + the ordinary "Could not resolve target" error and sends no input. + """ + if target_missed is None: + return self._resolve_step(step, screen_png, bundle_dir, workflow) + resolution, region, error = self._resolve_step( + step, screen_png, bundle_dir, workflow, allow_grounder=False + ) + if resolution is not None and resolution.rung in _DIRECT_TARGET_RUNGS: + return resolution, region, error + if resolution is None and step.anchor is None: + return resolution, region, error + return ( + None, + None, + f"Could not resolve target for step '{step.id}' ({step.intent}): " + f"{target_missed}", + ) + def _resolve_step( self, step: Step, @@ -15041,6 +15100,7 @@ def _implicit_scroll_target_ready( bundle_dir: Path, params: Mapping[str, RuntimeParamScalar], scroll_delta: Optional[tuple[int, int]] = None, + misses: Optional[list[str]] = None, ) -> bool: """Resolve an implicit scroll target and honor its armed identity. @@ -15057,9 +15117,19 @@ def _implicit_scroll_target_ready( the preceding scroll a false no-op. Structural resolution stays available, model grounding stays disabled, and an identity-armed target is ready only when the same pre-action identity gate verifies it. + + When not ready, ``misses`` (if given) gets why: ``unresolved`` (no + direct deterministic rung found the target), ``position`` or + ``identity``. """ - if step.anchor is None: + + def miss(reason: str) -> bool: + if misses is not None: + misses.append(reason) return False + + if step.anchor is None: + return miss("unresolved") template_png = self._asset_bytes( bundle_dir, step.anchor.template, workflow=workflow ) @@ -15084,9 +15154,9 @@ def _implicit_scroll_target_ready( # candidate means that the target is not ready yet, so the scroll # loop must continue. The later target action runs the full # resolver again and still halts on the same ambiguity. - return False + return miss("unresolved") if resolved is None: - return False + return miss("unresolved") resolution, _matched_region = resolved # Geometry estimates a point from surrounding landmarks; it does not # prove that the target itself is visible. In particular, a fixed @@ -15095,23 +15165,18 @@ def _implicit_scroll_target_ready( # readiness therefore requires direct deterministic target evidence. # Model grounding is disabled above and future weaker rungs must opt in # deliberately rather than silently acquiring readiness authority. - if resolution.rung not in { - "structural", - "template", - "template_global", - "ocr", - }: - return False + if resolution.rung not in _DIRECT_TARGET_RUNGS: + return miss("unresolved") if scroll_delta is not None and not _scroll_target_positioned( step.anchor.click_point, resolution.point, scroll_delta, frame_png ): - return False + return miss("position") if not step.identity_armed: return True check = self._verify_identity( step, resolution, frame_png, params, workflow, bundle_dir ) - return check.status == "verified" + return check.status == "verified" or miss("identity") def _act_scroll( self, @@ -15192,6 +15257,9 @@ def _act_scroll( # Until a wheel edge stops moving the page, readiness also requires # the target at (or past) its demonstrated position. position_required = [True] + # Why each readiness probe of the next target failed; they decide who + # owns a budget-exhaustion halt (see below). + misses: list[str] = [] def readiness_holds(frame_png: bytes) -> bool: if implicit_target_readiness: @@ -15203,6 +15271,7 @@ def readiness_holds(frame_png: bytes) -> bool: bundle_dir=bundle_dir, params=params, scroll_delta=(dx, dy) if position_required[0] else None, + misses=misses, ) return self._predicate_holds( stop_pred, @@ -15284,6 +15353,32 @@ def readiness_holds(frame_png: bytes) -> bool: if following is not None and following.action is ActionKind.SCROLL: # The next SCROLL step continues the loop with its own budget. return None + if ( + graph_ctx is None + and implicit_target_readiness + and next_step is not None + and following is next_step + and next_step.action in _SCROLL_HANDOFF_ACTIONS + and misses + and all(miss == "unresolved" for miss in misses) + and not step.risk_review_required + and not self._step_is_consequential(step, workflow) + ): + # The scroll did its bounded work; what failed is the next click's + # target, which never resolved on any frame (a relabelled or moved + # control). Halting here would name the scroll and hide the + # control that changed. Hand the miss to that click instead: it + # resolves on direct deterministic rungs only and, when they find + # nothing, halts on its own target before any input. A target + # that resolved but sat out of position or failed its identity + # check still halts here, as before. + self._scroll_missed_target = ( + next_step.id, + f"the preceding scroll '{step.id}' used {scrolled:.0f}px of its " + f"{budget:.0f}px budget and the target never came into view; " + "no deterministic rung found it, so no input was sent", + ) + return None target_desc = ( f"the anchor of step '{next_step.id}' ({next_step.intent})" if next_step is not None diff --git a/packages/flow/tests/test_relabelled_save_halt.py b/packages/flow/tests/test_relabelled_save_halt.py new file mode 100644 index 0000000000..0b5ee39174 --- /dev/null +++ b/packages/flow/tests/test_relabelled_save_halt.py @@ -0,0 +1,191 @@ +"""A relabelled Save behind a scroll halts on the Save, ready for repair (#1331). + +Synthetic fixture shaped like the openIMIS claim program: a key press and an +unlabelled picker click whose risk the compiler could not classify, then a +closed-loop scroll back up to an icon-only Save. The vendor relabels the Save, +so nothing on the deterministic ladder finds it. + +* The halt names the Save, not the scroll that was looking for it, and no + input reaches the Save. +* While the key press and the picker click are unreviewed, a write may have + happened, so the run ends RECONCILIATION_REQUIRED and self-repair refuses. +* Once a person reviews both as reversible (``qualify set-risk``), the same + halt is HALTED_BEFORE_EFFECT and self-repair takes the Save. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openadapt_flow.ir import ActionKind, Anchor, Step, Workflow +from openadapt_flow.qualification import ( + ActionRiskClass, + ActionRiskClassification, + EnvironmentBoundary, + init_project, + set_action_classification, +) +from openadapt_flow.repair.self_repair import _not_before_effect, _repairable +from openadapt_flow.runtime.replayer import Replayer +from openadapt_flow.transaction import ( + TransactionOutcome, + classify_transaction_outcome, + steps_requiring_reconciliation, +) +from tests.test_replayer import FakeBackend, FakeVision, Match, make_png + +PICKER_PNG = make_png((40, 30)) +SAVE_PNG = make_png((24, 24)) +PICKER = Match(point=(210, 305), region=(190, 290, 40, 30), confidence=0.95) + + +class RelabelledSaveVision(FakeVision): + """The picker is where it was; the recorded Save is on no frame.""" + + def find_template(self, screen_png, template_png, **kwargs): + self.template_calls.append(kwargs.get("search_region")) + return PICKER if template_png == PICKER_PNG else None + + +def _claim_program() -> Workflow: + return Workflow( + name="claim", + steps=[ + Step( + id="step_002", + intent="press Tab to look up the insurance number", + action=ActionKind.KEY, + key="Tab", + risk_review_required=True, + risk_explanation="special key may navigate, edit, or invoke a shortcut", + ), + Step( + id="step_012", + intent="click the service picker", + action=ActionKind.CLICK, + anchor=Anchor( + template="templates/picker.png", + region=(190, 290, 40, 30), + click_point=(210, 305), + ), + risk_review_required=True, + risk_explanation="unlabelled pointer control", + ), + Step( + id="step_014", + intent="scroll by (0, -600)", + action=ActionKind.SCROLL, + scroll_dx=0, + scroll_dy=-600, + ), + Step( + id="step_015", + intent="click the icon-only Save", + action=ActionKind.CLICK, + anchor=Anchor( + template="templates/save.png", + region=(600, 40, 24, 24), + click_point=(612, 52), + ), + risk="irreversible", + timeout_s=0.2, + ), + ], + ) + + +def _review_as_reversible(workflow: Workflow) -> None: + init_project( + workflow, + environment=EnvironmentBoundary( + target_kind="web", + application="synthetic-claims", + application_version="1", + environment_digest="0" * 64, + runtime_version="test", + ), + ) + for step_id in ("step_002", "step_012"): + set_action_classification( + workflow, + ActionRiskClassification( + step_id=step_id, + classification=ActionRiskClass.STATE_CHANGING, + explanation="moves focus or opens a list; saves nothing", + operator_confirmed=True, + ), + ) + + +@pytest.fixture() +def bundle(tmp_path) -> Path: + bundle_dir = tmp_path / "bundle" + (bundle_dir / "templates").mkdir(parents=True) + (bundle_dir / "templates" / "picker.png").write_bytes(PICKER_PNG) + (bundle_dir / "templates" / "save.png").write_bytes(SAVE_PNG) + return bundle_dir + + +def _run(workflow: Workflow, bundle: Path, run_dir: Path): + backend = FakeBackend() + report = Replayer(backend, vision=RelabelledSaveVision(), poll_interval_s=0.01).run( + workflow, bundle_dir=bundle, run_dir=run_dir + ) + return backend, report + + +def test_the_halt_names_the_save_not_the_scroll(bundle, tmp_path) -> None: + backend, report = _run(_claim_program(), bundle, tmp_path / "run") + + assert report.success is False + assert report.halt is not None and report.halt.state_id == "step_015" + by_id = {r.step_id: r for r in report.results} + assert by_id["step_014"].ok + save = by_id["step_015"] + assert save.delivery_attempted is False + assert save.resolution is None + assert save.error.startswith("Could not resolve target for step 'step_015'") + assert "step_014" in save.error + # Tab, the picker, two upward wheel edges (2.5x budget), and no Save click. + assert backend.actions == [ + ("press", "Tab"), + ("click", 210, 305, False), + ("scroll", 0, -600), + ("scroll", 0, -600), + ] + + +def test_unreviewed_risk_keeps_the_run_in_reconciliation(bundle, tmp_path) -> None: + workflow = _claim_program() + _, report = _run(workflow, bundle, tmp_path / "run") + + # A key press or click whose risk nobody reviewed may have written, and no + # verifier read says it did not: fail closed. + assert ( + classify_transaction_outcome(report) + is TransactionOutcome.RECONCILIATION_REQUIRED + ) + assert steps_requiring_reconciliation(report) == ["step_002", "step_012"] + # Self-repair refuses, and says which steps and who can settle them. + report.transaction_outcome = "RECONCILIATION_REQUIRED" + reason = _not_before_effect(report) + assert "effect not settled at step_002, step_012" in reason + assert "qualify set-risk" in reason + + +def test_reviewed_risk_makes_the_halt_repairable(bundle, tmp_path) -> None: + workflow = _claim_program() + _review_as_reversible(workflow) + _, report = _run(workflow, bundle, tmp_path / "run") + + assert report.halt is not None and report.halt.state_id == "step_015" + assert ( + classify_transaction_outcome(report) is TransactionOutcome.HALTED_BEFORE_EFFECT + ) + assert steps_requiring_reconciliation(report) == [] + found = _repairable(report, workflow) + assert not isinstance(found, str), found + step, index, halted = found + assert (step.id, index, halted.step_id) == ("step_015", 3, "step_015") diff --git a/packages/flow/tests/test_replayer.py b/packages/flow/tests/test_replayer.py index c8bccd5aab..7af45bc11c 100644 --- a/packages/flow/tests/test_replayer.py +++ b/packages/flow/tests/test_replayer.py @@ -2309,6 +2309,7 @@ def test_closed_loop_scroll_does_not_accept_geometry_as_target_visibility( ) ], ) + target.timeout_s = 0.2 workflow = Workflow(name="wf", steps=[scroll_step(), target]) report = Replayer(backend := FakeBackend(), vision=vision).run( @@ -2316,33 +2317,76 @@ def test_closed_loop_scroll_does_not_accept_geometry_as_target_visibility( ) assert report.success is False - assert "target never came into view" in report.results[0].error + # The click owns the halt and refuses its geometry-only point: no click. + assert report.results[0].ok + halted = report.results[-1] + assert halted.step_id == target.id + assert halted.resolution is None + assert halted.delivery_attempted is False + assert "target never came into view" in halted.error assert backend.actions == [("scroll", 0, 400), ("scroll", 0, 400)] def test_closed_loop_scroll_budget_exhaustion_fails_loudly(bundle, run_dir): """When the anchor never resolves and no further SCROLL step follows, - the loop stops at ~2.5x the recorded distance and fails the run, - naming the anchor that never came into view.""" + the loop stops at ~2.5x the recorded distance and the run halts on the + click whose target never came into view (#1331), before any input to it. + """ vision = FakeVision() # every probe misses backend = FakeBackend() - workflow = Workflow( - name="wf", - steps=[scroll_step(dy=-400), click_step(step_id="pencil")], - ) + pencil = click_step(step_id="pencil") + pencil.timeout_s = 0.2 + workflow = Workflow(name="wf", steps=[scroll_step(dy=-400), pencil]) report = Replayer(backend, vision=vision, poll_interval_s=0.01).run( workflow, bundle_dir=bundle, run_dir=run_dir ) assert report.success is False - assert len(report.results) == 1 # aborted at the scroll step - error = report.results[0].error - assert "budget" in error - assert "pencil" in error # names the anchor that never resolved + assert len(report.results) == 2 + assert report.results[0].ok + halted = report.results[1] + assert halted.step_id == "pencil" + assert halted.delivery_attempted is False + assert halted.error.startswith("Could not resolve target for step 'pencil'") + assert "budget" in halted.error + assert report.halt is not None and report.halt.state_id == "pencil" # Budget 2.5 x 400px allows exactly two 400px gestures, both upward # (direction comes from the recorded delta). assert backend.actions == [("scroll", 0, -400), ("scroll", 0, -400)] +def test_scroll_keeps_the_halt_when_the_target_resolves_but_fails_identity( + bundle, run_dir, monkeypatch +): + """Only a target no direct rung found is handed to the click (#1331). + + A target that resolves on every frame but never verifies its record + identity is not a changed control: the scroll keeps the halt and the click + never runs. + """ + vision = FakeVision() + target = Match(point=(110, 105), region=(100, 100, 50, 20), confidence=0.99) + vision.template_results = [target] * 6 + backend = FakeBackend() + click = click_step() + assert click.anchor is not None + click.anchor.context_text = "Synthetic Contact Address" + click.identity_armed = True + workflow = Workflow(name="wf", steps=[scroll_step(), click]) + replayer = Replayer(backend, vision=vision, poll_interval_s=0.01) + monkeypatch.setattr( + replayer, + "_verify_identity", + lambda *args, **kwargs: IdentityCheck( + status="mismatch", expected="target", observed="wrong" + ), + ) + report = replayer.run(workflow, bundle_dir=bundle, run_dir=run_dir) + assert report.success is False + assert len(report.results) == 1 + assert "target never came into view" in report.results[0].error + assert backend.actions == [("scroll", 0, 400), ("scroll", 0, 400)] + + def test_consequential_remote_scroll_reacquires_each_wheel_edge(bundle, run_dir): frame = make_png() backend = RemoteLeaseBackend(initial_frame=frame, fresh_frame=frame) From 62b07477a8d5ca19e0a1a3f0c881f8d5e578d3e6 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 09:53:17 -0400 Subject: [PATCH 2/2] docs: point the capability row at #1337 Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- llms-full.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 93ab08cd60..553017c644 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -189,7 +189,7 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1331](https://github.com/OpenAdaptAI/OpenAdapt/issues/1331)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | | Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | diff --git a/llms-full.txt b/llms-full.txt index 3bbedf8001..e216774de4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -545,7 +545,7 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1331](https://github.com/OpenAdaptAI/OpenAdapt/issues/1331)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | | Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes |