diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 2768c8b4f8..f0bd799894 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -200,6 +200,7 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | | Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | diff --git a/llms-full.txt b/llms-full.txt index a2cf209b6b..908526fd2e 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -263,6 +263,7 @@ including the few still being merged. | Stopped before saving | `HALTED_BEFORE_EFFECT` | `needs_review` | Fix the cause, or show the program the fix, and run again. | | Check the record | `RECONCILIATION_REQUIRED` | `not_sure_if_saved` | A save may have gone through. A person checks the record before anything is retried. | | Finished, not checked | `COMPLETED_UNVERIFIED` | `not_sure_if_saved` | Fine for testing; not a production success. | +| Saved, waiting for its check | `SAVED_AWAITING_CHECK` | `not_sure_if_saved` | The save's check is a later report (a claim acknowledgment, an export). Don't rerun it. `openadapt-flow deferred settle` counts it as done only when the report confirms it. | | Didn't run | `REJECTED_POLICY`, `CANCELED`, `FAILED_PLATFORM` | `did_not_run` | Fix the gate or the environment and run again. | The console's last line can read `HALTED` for both the second and third rows. @@ -415,7 +416,7 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is landing. + without saving, is on main; fixes of up to three new steps are in review. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` @@ -434,8 +435,13 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | +| Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | +| Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | +| Lists every change made before a stop, and reverses them with undo steps a person approves. | `undo`, [AFTER_A_STOP.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/AFTER_A_STOP.md) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | +| Repairs a change that needs new steps, such as a new required dialog or a control moved into a menu: up to three new steps, each with its own record check, probed up to the save before one person approves. | [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md), [#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | | The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | @@ -550,10 +556,11 @@ improvising or retrying when something is off. | Governed repair: propose, review, campaigns, approval, staging, canary, rollback | Released | Synthetic | `openadapt-flow repair`, [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | -| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | +| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | -| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | +| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents @@ -584,9 +591,9 @@ improvising or retrying when something is off. Each `openadapt-flow` command, grouped by the capability it serves. A test fails when a command is added without being listed here. -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`. +- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. - **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. +- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). - **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). - **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. @@ -614,7 +621,9 @@ fails when a command is added without being listed here. ## What is not done yet - No commercial EMR, claims system or real Citrix session has been tested. -- In progress: self-repair across several new steps, and the scoreboard. +- In review: self-repair across several new steps + ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In + progress: the scoreboard. - Windows installers are unsigned until free code signing through SignPath Foundation is approved; macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The @@ -1347,7 +1356,10 @@ record in the acting session (tier 3) is reported as its own check level, confirmed reopen ends `COMPLETED_UNVERIFIED`, and the agent result says `not_sure_if_saved` with `checked_below_minimum` ([`READBACK_LADDER.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/READBACK_LADDER.md)). A workflow's `minimum_readback` -can raise the floor to `system_of_record`, never lower it. The on-screen +can raise the floor to `system_of_record`, never lower it. +A save that only a later report can confirm (a claim acknowledgment, a +nightly export) ends `SAVED_AWAITING_CHECK` and becomes `VERIFIED` only when +that report accepts it ([`deferred_report`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#checked-later-deferred_report)). The on-screen oracle can still halt when the region is unreadable or wrong. If a consequential write may already have been delivered and the check did not confirm it, the production `transaction_outcome` is `RECONCILIATION_REQUIRED`, diff --git a/packages/flow/docs/SELF_REPAIR.md b/packages/flow/docs/SELF_REPAIR.md index 90e2e3c8b8..58026f1abb 100644 --- a/packages/flow/docs/SELF_REPAIR.md +++ b/packages/flow/docs/SELF_REPAIR.md @@ -31,6 +31,19 @@ Module: `openadapt_flow.repair.self_repair`. CLI: `openadapt-flow run A halt on the record check (an identity mismatch) is never repaired, and neither is an item a person declined at the approval gate. + + When a closed-loop scroll runs out of budget because the next click's + target never resolved, the halt is on that click, not on the scroll. The + click accepts only direct rungs (page structure, template, OCR) and no + model, so it sends no input. A target that resolved but was out of place or + failed its record check still halts on the scroll. + + A run that ends `RECONCILIATION_REQUIRED` is never repaired. A key press or + click whose risk nobody reviewed (`risk_review_required`) may have written, + so the run ends that way even when the halt itself sent nothing. The + refusal names those steps. A person who reviews them with + `openadapt-flow qualify set-risk` makes the same halt + `HALTED_BEFORE_EFFECT`, and self-repair can take it. 2. **Propose.** The repair agent is shown the step's intent, the recorded evidence (the target crop, its label, its place), the live screen, the halt kind, and for `after` the text of the screen the step expected. It diff --git a/packages/flow/openadapt_flow/benchmark/reliability.py b/packages/flow/openadapt_flow/benchmark/reliability.py index 605788d3da..8deddb4e90 100644 --- a/packages/flow/openadapt_flow/benchmark/reliability.py +++ b/packages/flow/openadapt_flow/benchmark/reliability.py @@ -289,7 +289,12 @@ def classify_failure(report: Optional[RunReport], error: Optional[str]) -> dict: return {"category": "none", "root_cause_hint": None} err = (failing.error or "").lower() - if failing.resolution is None and "all resolution rungs failed" in err: + if failing.resolution is None and ( + "all resolution rungs failed" in err + # A closed-loop scroll that never found the target hands the halt to + # that target's step (#1331). + or err.startswith("could not resolve target for step") + ): category = "resolution_failed" elif "identity check failed" in err: category = "identity_mismatch" diff --git a/packages/flow/openadapt_flow/repair/self_repair.py b/packages/flow/openadapt_flow/repair/self_repair.py index 2f5a221b93..8e0e6f95fb 100644 --- a/packages/flow/openadapt_flow/repair/self_repair.py +++ b/packages/flow/openadapt_flow/repair/self_repair.py @@ -697,8 +697,7 @@ def run(self, bundle_dir: Path | str, run_dir: Path | str) -> SelfRepairResult: if report.transaction_outcome not in (None, "HALTED_BEFORE_EFFECT"): return self._finish( "not_repairable", - f"the run ended {report.transaction_outcome}; self-repair " - "handles only a halt before any effect", + _not_before_effect(report), evidence, report, run_dir, @@ -2152,6 +2151,31 @@ def _recorded_crop(workflow: Workflow, bundle_dir: Path, step: Step) -> Optional return path.read_bytes() if path.is_file() else None +def _not_before_effect(report: RunReport) -> str: + """Why a halt that may have left an effect is not repaired, and by whom.""" + + from openadapt_flow.transaction import steps_requiring_reconciliation + + reason = ( + f"the run ended {report.transaction_outcome}; self-repair handles only " + "a halt before any effect" + ) + unsettled = steps_requiring_reconciliation(report) + if not unsettled: + return reason + reason += f" (effect not settled at {', '.join(unsettled[:8])})" + unreviewed = {r.step_id for r in report.results if r.risk_review_required} + if unreviewed.issuperset(unsettled): + # Only steps nobody classified keep this run from an absence claim. + # A person reviews them once; the rule itself stays fail-closed. + reason += ( + "; each acted with a risk nobody reviewed, so a write cannot be " + "ruled out. A person can classify them with `openadapt-flow " + "qualify set-risk`" + ) + return reason[:1024] + + def _repairable(report: RunReport, workflow: Workflow) -> "tuple[Step, str, Any] | str": """``(step, mode, result)`` when self-repair may handle this halt, else why not. diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index 6dcfdf5fec..02017174c3 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -577,6 +577,16 @@ class EgressNotPermitted(RuntimeError): #: scroll counts it as ready. SCROLL_POSITION_SLACK_FRACTION = 0.25 +#: Rungs that show the target itself on screen: what closed-loop scroll +#: readiness accepts, and all a step accepts after its scroll missed it. +_DIRECT_TARGET_RUNGS = frozenset({"structural", "template", "template_global", "ocr"}) + +#: Steps that own the halt when the closed-loop scroll before them never found +#: their target (see ``Replayer._act_scroll``). +_SCROLL_HANDOFF_ACTIONS = frozenset( + {ActionKind.CLICK, ActionKind.DOUBLE_CLICK, ActionKind.RIGHT_CLICK} +) + def _scroll_target_positioned( recorded: tuple[int, int], @@ -1047,6 +1057,11 @@ def __init__( self.web_form_actuator = web_form_actuator self._web_form_span_active = False self._web_form_fallback_reason: Optional[str] = None + # (step id, reason) set when a closed-loop scroll used up its budget + # because the next step's target never resolved. That step then + # resolves on direct deterministic rungs only, and halts on its own + # target, so the halt names the control that changed, not the scroll. + self._scroll_missed_target: Optional[tuple[str, str]] = None # Under an approval mode, the API request a person was asked about # (prepared before the gate), keyed by the save step's result, and # the operation digest the gate's last-point check re-reads. @@ -1597,6 +1612,7 @@ def run( else uuid.uuid4().hex ) self._field_reads = {} + self._scroll_missed_target = None self._idempotency_reservation_owned = False self._qualification_fault_mutations = [] self._qualification_fault_public_key = None @@ -6903,6 +6919,13 @@ def _run_step_timed( failure_category="safety_halt", ) return refused + # A closed-loop scroll that never found this step's target hands the + # miss to this step (see ``_act_scroll``); only this step consumes it. + target_missed: Optional[str] = None + missed = self._scroll_missed_target + if missed is not None and missed[0] == step.id: + self._scroll_missed_target = None + target_missed = missed[1] router = self._surface_router if router is None: if step.action is ActionKind.SWITCH_SURFACE: @@ -6920,6 +6943,7 @@ def _run_step_timed( run_dir=run_dir, new_crops=new_crops, graph_ctx=graph_ctx, + target_missed=target_missed, ) refusal = router.entry_refusal(step) if refusal is None and step.action is not ActionKind.SWITCH_SURFACE: @@ -6952,6 +6976,7 @@ def _run_step_timed( run_dir=run_dir, new_crops=new_crops, graph_ctx=graph_ctx, + target_missed=target_missed, ) # Receipt: the surface this step ran on. The entry check above proved # it is the step's declared surface; the outcome classifier re-checks @@ -7066,6 +7091,7 @@ def _run_step_on_active_surface( run_dir: Path, new_crops: dict[str, bytes], graph_ctx: Optional["_GraphStepContext"] = None, + target_missed: Optional[str] = None, ) -> StepResult: """Execute a single step; never raises (failures land in the result). @@ -7398,8 +7424,8 @@ def _run_step_on_active_surface( ) result.elapsed_ms = (time.monotonic() - t0) * 1000.0 return result - resolution, matched_region, error = self._resolve_step( - step, before_png, bundle_dir, workflow + resolution, matched_region, error = self._resolve_step_target( + step, before_png, bundle_dir, workflow, target_missed ) # Retry ladder failures with fresh settled frames until # ``step.timeout_s``: a remote app can present a settled-looking @@ -7419,8 +7445,8 @@ def _run_step_on_active_surface( run_dir, evidence_step_id, "before", before_png ) last_frame = before_png - resolution, matched_region, error = self._resolve_step( - step, before_png, bundle_dir, workflow + resolution, matched_region, error = self._resolve_step_target( + step, before_png, bundle_dir, workflow, target_missed ) if resolution is not None and matched_region is not None: assert step.anchor is not None @@ -10735,6 +10761,39 @@ def _revalidate_consequential_actuation( self._active_delivery_region = fresh_region return fresh_resolution, fresh_region, fresh_png, error + def _resolve_step_target( + self, + step: Step, + screen_png: bytes, + bundle_dir: Path, + workflow: Workflow, + target_missed: Optional[str], + ) -> tuple[Optional[Resolution], Optional[Region], Optional[str]]: + """Resolve the step's target; stricter after a scroll that missed it. + + ``target_missed`` is set when the preceding closed-loop scroll used + up its budget without this target resolving. The step then accepts + only the rungs scroll readiness accepts (direct deterministic + evidence, no model): a weaker rung would act where the scroll could + not find the target. Otherwise the step halts on its own target with + the ordinary "Could not resolve target" error and sends no input. + """ + if target_missed is None: + return self._resolve_step(step, screen_png, bundle_dir, workflow) + resolution, region, error = self._resolve_step( + step, screen_png, bundle_dir, workflow, allow_grounder=False + ) + if resolution is not None and resolution.rung in _DIRECT_TARGET_RUNGS: + return resolution, region, error + if resolution is None and step.anchor is None: + return resolution, region, error + return ( + None, + None, + f"Could not resolve target for step '{step.id}' ({step.intent}): " + f"{target_missed}", + ) + @timed_span("resolve") def _resolve_step( self, @@ -15504,6 +15563,7 @@ def _implicit_scroll_target_ready( bundle_dir: Path, params: Mapping[str, RuntimeParamScalar], scroll_delta: Optional[tuple[int, int]] = None, + misses: Optional[list[str]] = None, ) -> bool: """Resolve an implicit scroll target and honor its armed identity. @@ -15520,9 +15580,20 @@ def _implicit_scroll_target_ready( the preceding scroll a false no-op. Structural resolution stays available, model grounding stays disabled, and an identity-armed target is ready only when the same pre-action identity gate verifies it. + + When not ready, ``misses`` (if given) gets why: ``unresolved`` (no + direct deterministic rung found the target), ``offscreen`` (the page + reports the recorded element outside the viewport), ``position`` or + ``identity``. """ - if step.anchor is None: + + def miss(reason: str) -> bool: + if misses is not None: + misses.append(reason) return False + + if step.anchor is None: + return miss("unresolved") structural = ( self.backend if self.use_structural and hasattr(self.backend, "locate_structural") @@ -15537,8 +15608,9 @@ def _implicit_scroll_target_ready( # The page itself shows the unique recorded target exists below # (or above) the viewport: not ready, keep scrolling. Stricter # than letting a visual rung find a look-alike on screen, and it - # skips a full-frame OCR search per probe. - return False + # skips a full-frame OCR search per probe. The control exists, so + # this miss never hands the halt to the click (#1331). + return miss("offscreen") template_png = self._asset_bytes( bundle_dir, step.anchor.template, workflow=workflow ) @@ -15558,9 +15630,9 @@ def _implicit_scroll_target_ready( # candidate means that the target is not ready yet, so the scroll # loop must continue. The later target action runs the full # resolver again and still halts on the same ambiguity. - return False + return miss("unresolved") if resolved is None: - return False + return miss("unresolved") resolution, _matched_region = resolved # Geometry estimates a point from surrounding landmarks; it does not # prove that the target itself is visible. In particular, a fixed @@ -15569,23 +15641,18 @@ def _implicit_scroll_target_ready( # readiness therefore requires direct deterministic target evidence. # Model grounding is disabled above and future weaker rungs must opt in # deliberately rather than silently acquiring readiness authority. - if resolution.rung not in { - "structural", - "template", - "template_global", - "ocr", - }: - return False + if resolution.rung not in _DIRECT_TARGET_RUNGS: + return miss("unresolved") if scroll_delta is not None and not _scroll_target_positioned( step.anchor.click_point, resolution.point, scroll_delta, frame_png ): - return False + return miss("position") if not step.identity_armed: return True check = self._verify_identity( step, resolution, frame_png, params, workflow, bundle_dir ) - return check.status == "verified" + return check.status == "verified" or miss("identity") def _act_scroll( self, @@ -15666,6 +15733,9 @@ def _act_scroll( # Until a wheel edge stops moving the page, readiness also requires # the target at (or past) its demonstrated position. position_required = [True] + # Why each readiness probe of the next target failed; they decide who + # owns a budget-exhaustion halt (see below). + misses: list[str] = [] def readiness_holds(frame_png: bytes) -> bool: if implicit_target_readiness: @@ -15677,6 +15747,7 @@ def readiness_holds(frame_png: bytes) -> bool: bundle_dir=bundle_dir, params=params, scroll_delta=(dx, dy) if position_required[0] else None, + misses=misses, ) return self._predicate_holds( stop_pred, @@ -15758,6 +15829,32 @@ def readiness_holds(frame_png: bytes) -> bool: if following is not None and following.action is ActionKind.SCROLL: # The next SCROLL step continues the loop with its own budget. return None + if ( + graph_ctx is None + and implicit_target_readiness + and next_step is not None + and following is next_step + and next_step.action in _SCROLL_HANDOFF_ACTIONS + and misses + and all(miss == "unresolved" for miss in misses) + and not step.risk_review_required + and not self._step_is_consequential(step, workflow) + ): + # The scroll did its bounded work; what failed is the next click's + # target, which never resolved on any frame (a relabelled or moved + # control). Halting here would name the scroll and hide the + # control that changed. Hand the miss to that click instead: it + # resolves on direct deterministic rungs only and, when they find + # nothing, halts on its own target before any input. A target + # that resolved but sat out of position or failed its identity + # check still halts here, as before. + self._scroll_missed_target = ( + next_step.id, + f"the preceding scroll '{step.id}' used {scrolled:.0f}px of its " + f"{budget:.0f}px budget and the target never came into view; " + "no deterministic rung found it, so no input was sent", + ) + return None target_desc = ( f"the anchor of step '{next_step.id}' ({next_step.intent})" if next_step is not None diff --git a/packages/flow/tests/test_relabelled_save_halt.py b/packages/flow/tests/test_relabelled_save_halt.py new file mode 100644 index 0000000000..fd94679330 --- /dev/null +++ b/packages/flow/tests/test_relabelled_save_halt.py @@ -0,0 +1,191 @@ +"""A relabelled Save behind a scroll halts on the Save, ready for repair (#1331). + +Synthetic fixture shaped like the openIMIS claim program: a key press and an +unlabelled picker click whose risk the compiler could not classify, then a +closed-loop scroll back up to an icon-only Save. The vendor relabels the Save, +so nothing on the deterministic ladder finds it. + +* The halt names the Save, not the scroll that was looking for it, and no + input reaches the Save. +* While the key press and the picker click are unreviewed, a write may have + happened, so the run ends RECONCILIATION_REQUIRED and self-repair refuses. +* Once a person reviews both as reversible (``qualify set-risk``), the same + halt is HALTED_BEFORE_EFFECT and self-repair takes the Save. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openadapt_flow.ir import ActionKind, Anchor, Step, Workflow +from openadapt_flow.qualification import ( + ActionRiskClass, + ActionRiskClassification, + EnvironmentBoundary, + init_project, + set_action_classification, +) +from openadapt_flow.repair.self_repair import _not_before_effect, _repairable +from openadapt_flow.runtime.replayer import Replayer +from openadapt_flow.transaction import ( + TransactionOutcome, + classify_transaction_outcome, + steps_requiring_reconciliation, +) +from tests.test_replayer import FakeBackend, FakeVision, Match, make_png + +PICKER_PNG = make_png((40, 30)) +SAVE_PNG = make_png((24, 24)) +PICKER = Match(point=(210, 305), region=(190, 290, 40, 30), confidence=0.95) + + +class RelabelledSaveVision(FakeVision): + """The picker is where it was; the recorded Save is on no frame.""" + + def find_template(self, screen_png, template_png, **kwargs): + self.template_calls.append(kwargs.get("search_region")) + return PICKER if template_png == PICKER_PNG else None + + +def _claim_program() -> Workflow: + return Workflow( + name="claim", + steps=[ + Step( + id="step_002", + intent="press Tab to look up the insurance number", + action=ActionKind.KEY, + key="Tab", + risk_review_required=True, + risk_explanation="special key may navigate, edit, or invoke a shortcut", + ), + Step( + id="step_012", + intent="click the service picker", + action=ActionKind.CLICK, + anchor=Anchor( + template="templates/picker.png", + region=(190, 290, 40, 30), + click_point=(210, 305), + ), + risk_review_required=True, + risk_explanation="unlabelled pointer control", + ), + Step( + id="step_014", + intent="scroll by (0, -600)", + action=ActionKind.SCROLL, + scroll_dx=0, + scroll_dy=-600, + ), + Step( + id="step_015", + intent="click the icon-only Save", + action=ActionKind.CLICK, + anchor=Anchor( + template="templates/save.png", + region=(600, 40, 24, 24), + click_point=(612, 52), + ), + risk="irreversible", + timeout_s=0.2, + ), + ], + ) + + +def _review_as_reversible(workflow: Workflow) -> None: + init_project( + workflow, + environment=EnvironmentBoundary( + target_kind="web", + application="synthetic-claims", + application_version="1", + environment_digest="0" * 64, + runtime_version="test", + ), + ) + for step_id in ("step_002", "step_012"): + set_action_classification( + workflow, + ActionRiskClassification( + step_id=step_id, + classification=ActionRiskClass.STATE_CHANGING, + explanation="moves focus or opens a list; saves nothing", + operator_confirmed=True, + ), + ) + + +@pytest.fixture() +def bundle(tmp_path) -> Path: + bundle_dir = tmp_path / "bundle" + (bundle_dir / "templates").mkdir(parents=True) + (bundle_dir / "templates" / "picker.png").write_bytes(PICKER_PNG) + (bundle_dir / "templates" / "save.png").write_bytes(SAVE_PNG) + return bundle_dir + + +def _run(workflow: Workflow, bundle: Path, run_dir: Path): + backend = FakeBackend() + report = Replayer(backend, vision=RelabelledSaveVision(), poll_interval_s=0.01).run( + workflow, bundle_dir=bundle, run_dir=run_dir + ) + return backend, report + + +def test_the_halt_names_the_save_not_the_scroll(bundle, tmp_path) -> None: + backend, report = _run(_claim_program(), bundle, tmp_path / "run") + + assert report.success is False + assert report.halt is not None and report.halt.state_id == "step_015" + by_id = {r.step_id: r for r in report.results} + assert by_id["step_014"].ok + save = by_id["step_015"] + assert save.delivery_attempted is False + assert save.resolution is None + assert save.error.startswith("Could not resolve target for step 'step_015'") + assert "step_014" in save.error + # Tab, the picker, two upward wheel edges (2.5x budget), and no Save click. + assert backend.actions == [ + ("press", "Tab"), + ("click", 210, 305, False), + ("scroll", 0, -600), + ("scroll", 0, -600), + ] + + +def test_unreviewed_risk_keeps_the_run_in_reconciliation(bundle, tmp_path) -> None: + workflow = _claim_program() + _, report = _run(workflow, bundle, tmp_path / "run") + + # A key press or click whose risk nobody reviewed may have written, and no + # verifier read says it did not: fail closed. + assert ( + classify_transaction_outcome(report) + is TransactionOutcome.RECONCILIATION_REQUIRED + ) + assert steps_requiring_reconciliation(report) == ["step_002", "step_012"] + # Self-repair refuses, and says which steps and who can settle them. + report.transaction_outcome = "RECONCILIATION_REQUIRED" + reason = _not_before_effect(report) + assert "effect not settled at step_002, step_012" in reason + assert "qualify set-risk" in reason + + +def test_reviewed_risk_makes_the_halt_repairable(bundle, tmp_path) -> None: + workflow = _claim_program() + _review_as_reversible(workflow) + _, report = _run(workflow, bundle, tmp_path / "run") + + assert report.halt is not None and report.halt.state_id == "step_015" + assert ( + classify_transaction_outcome(report) is TransactionOutcome.HALTED_BEFORE_EFFECT + ) + assert steps_requiring_reconciliation(report) == [] + found = _repairable(report, workflow) + assert not isinstance(found, str), found + step, mode, halted = found + assert (step.id, mode, halted.step_id) == ("step_015", "replace", "step_015") diff --git a/packages/flow/tests/test_replayer.py b/packages/flow/tests/test_replayer.py index c8bccd5aab..7af45bc11c 100644 --- a/packages/flow/tests/test_replayer.py +++ b/packages/flow/tests/test_replayer.py @@ -2309,6 +2309,7 @@ def test_closed_loop_scroll_does_not_accept_geometry_as_target_visibility( ) ], ) + target.timeout_s = 0.2 workflow = Workflow(name="wf", steps=[scroll_step(), target]) report = Replayer(backend := FakeBackend(), vision=vision).run( @@ -2316,33 +2317,76 @@ def test_closed_loop_scroll_does_not_accept_geometry_as_target_visibility( ) assert report.success is False - assert "target never came into view" in report.results[0].error + # The click owns the halt and refuses its geometry-only point: no click. + assert report.results[0].ok + halted = report.results[-1] + assert halted.step_id == target.id + assert halted.resolution is None + assert halted.delivery_attempted is False + assert "target never came into view" in halted.error assert backend.actions == [("scroll", 0, 400), ("scroll", 0, 400)] def test_closed_loop_scroll_budget_exhaustion_fails_loudly(bundle, run_dir): """When the anchor never resolves and no further SCROLL step follows, - the loop stops at ~2.5x the recorded distance and fails the run, - naming the anchor that never came into view.""" + the loop stops at ~2.5x the recorded distance and the run halts on the + click whose target never came into view (#1331), before any input to it. + """ vision = FakeVision() # every probe misses backend = FakeBackend() - workflow = Workflow( - name="wf", - steps=[scroll_step(dy=-400), click_step(step_id="pencil")], - ) + pencil = click_step(step_id="pencil") + pencil.timeout_s = 0.2 + workflow = Workflow(name="wf", steps=[scroll_step(dy=-400), pencil]) report = Replayer(backend, vision=vision, poll_interval_s=0.01).run( workflow, bundle_dir=bundle, run_dir=run_dir ) assert report.success is False - assert len(report.results) == 1 # aborted at the scroll step - error = report.results[0].error - assert "budget" in error - assert "pencil" in error # names the anchor that never resolved + assert len(report.results) == 2 + assert report.results[0].ok + halted = report.results[1] + assert halted.step_id == "pencil" + assert halted.delivery_attempted is False + assert halted.error.startswith("Could not resolve target for step 'pencil'") + assert "budget" in halted.error + assert report.halt is not None and report.halt.state_id == "pencil" # Budget 2.5 x 400px allows exactly two 400px gestures, both upward # (direction comes from the recorded delta). assert backend.actions == [("scroll", 0, -400), ("scroll", 0, -400)] +def test_scroll_keeps_the_halt_when_the_target_resolves_but_fails_identity( + bundle, run_dir, monkeypatch +): + """Only a target no direct rung found is handed to the click (#1331). + + A target that resolves on every frame but never verifies its record + identity is not a changed control: the scroll keeps the halt and the click + never runs. + """ + vision = FakeVision() + target = Match(point=(110, 105), region=(100, 100, 50, 20), confidence=0.99) + vision.template_results = [target] * 6 + backend = FakeBackend() + click = click_step() + assert click.anchor is not None + click.anchor.context_text = "Synthetic Contact Address" + click.identity_armed = True + workflow = Workflow(name="wf", steps=[scroll_step(), click]) + replayer = Replayer(backend, vision=vision, poll_interval_s=0.01) + monkeypatch.setattr( + replayer, + "_verify_identity", + lambda *args, **kwargs: IdentityCheck( + status="mismatch", expected="target", observed="wrong" + ), + ) + report = replayer.run(workflow, bundle_dir=bundle, run_dir=run_dir) + assert report.success is False + assert len(report.results) == 1 + assert "target never came into view" in report.results[0].error + assert backend.actions == [("scroll", 0, 400), ("scroll", 0, 400)] + + def test_consequential_remote_scroll_reacquires_each_wheel_edge(bundle, run_dir): frame = make_png() backend = RemoteLeaseBackend(initial_frame=frame, fresh_frame=frame)