Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ improvising or retrying when something is off.
| A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` |
| Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` |
| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) |
| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` |
| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) |
| Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) |
| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes |
Expand Down
30 changes: 21 additions & 9 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@ including the few still being merged.
| Stopped before saving | `HALTED_BEFORE_EFFECT` | `needs_review` | Fix the cause, or show the program the fix, and run again. |
| Check the record | `RECONCILIATION_REQUIRED` | `not_sure_if_saved` | A save may have gone through. A person checks the record before anything is retried. |
| Finished, not checked | `COMPLETED_UNVERIFIED` | `not_sure_if_saved` | Fine for testing; not a production success. |
| Saved, waiting for its check | `SAVED_AWAITING_CHECK` | `not_sure_if_saved` | The save's check is a later report (a claim acknowledgment, an export). Don't rerun it. `openadapt-flow deferred settle` counts it as done only when the report confirms it. |
| Didn't run | `REJECTED_POLICY`, `CANCELED`, `FAILED_PLATFORM` | `did_not_run` | Fix the gate or the environment and run again. |

The console's last line can read `HALTED` for both the second and third rows.
Expand Down Expand Up @@ -415,7 +416,7 @@ shipping in the next release; a pull request number means it is being merged.
review, campaign, canary, rollback), agent-demonstrated fixes for a halted
run, and fleet drift patches shared with no patient data are released.
Self-repair, where an agent proposes a fix and Flow probes it up to the save
without saving, is landing.
without saving, is on main; fixes of up to three new steps are in review.
9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded
save around any agent's write
([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run`
Expand All @@ -434,8 +435,13 @@ shipping in the next release; a pull request number means it is being merged.
| Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) |
| Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) |
| Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) |
| Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) |
| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) |
| Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) |
| Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) |
| Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) |
| Lists every change made before a stop, and reverses them with undo steps a person approves. | `undo`, [AFTER_A_STOP.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/AFTER_A_STOP.md) |
| Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) |
| Repairs a change that needs new steps, such as a new required dialog or a control moved into a menu: up to three new steps, each with its own record check, probed up to the save before one person approves. | [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md), [#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325) |
| Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) |
| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) |
| Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) |
Expand Down Expand Up @@ -550,10 +556,11 @@ improvising or retrying when something is off.
| Governed repair: propose, review, campaigns, approval, staging, canary, rollback | Released | Synthetic | `openadapt-flow repair`, [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) |
| A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` |
| Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` |
| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) |
| Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 |
| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) |
| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` |
| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) |
| Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) |
| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes |
| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes |
| Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) |

## Computer-use agents
Expand Down Expand Up @@ -584,9 +591,9 @@ improvising or retrying when something is off.
Each `openadapt-flow` command, grouped by the capability it serves. A test
fails when a command is added without being listed here.

- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`.
- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`.
- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`.
- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`.
- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`.
- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record).
- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop).
- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`.
Expand Down Expand Up @@ -614,7 +621,9 @@ fails when a command is added without being listed here.
## What is not done yet

- No commercial EMR, claims system or real Citrix session has been tested.
- In progress: self-repair across several new steps, and the scoreboard.
- In review: self-repair across several new steps
([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In
progress: the scoreboard.
- Windows installers are unsigned until free code signing through SignPath
Foundation is approved; macOS builds are ad-hoc signed.
- Several capabilities have run only on OpenAdapt's synthetic apps. The
Expand Down Expand Up @@ -1347,7 +1356,10 @@ record in the acting session (tier 3) is reported as its own check level,
confirmed reopen ends `COMPLETED_UNVERIFIED`, and the agent result says
`not_sure_if_saved` with `checked_below_minimum`
([`READBACK_LADDER.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/READBACK_LADDER.md)). A workflow's `minimum_readback`
can raise the floor to `system_of_record`, never lower it. The on-screen
can raise the floor to `system_of_record`, never lower it.
A save that only a later report can confirm (a claim acknowledgment, a
nightly export) ends `SAVED_AWAITING_CHECK` and becomes `VERIFIED` only when
that report accepts it ([`deferred_report`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#checked-later-deferred_report)). The on-screen
oracle can still halt when the region is unreadable or wrong. If a
consequential write may already have been delivered and the check did not
confirm it, the production `transaction_outcome` is `RECONCILIATION_REQUIRED`,
Expand Down
13 changes: 13 additions & 0 deletions packages/flow/docs/SELF_REPAIR.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ Module: `openadapt_flow.repair.self_repair`. CLI: `openadapt-flow run

A halt on the record check (an identity mismatch) is never repaired, and
neither is an item a person declined at the approval gate.

When a closed-loop scroll runs out of budget because the next click's
target never resolved, the halt is on that click, not on the scroll. The
click accepts only direct rungs (page structure, template, OCR) and no
model, so it sends no input. A target that resolved but was out of place or
failed its record check still halts on the scroll.

A run that ends `RECONCILIATION_REQUIRED` is never repaired. A key press or
click whose risk nobody reviewed (`risk_review_required`) may have written,
so the run ends that way even when the halt itself sent nothing. The
refusal names those steps. A person who reviews them with
`openadapt-flow qualify set-risk` makes the same halt
`HALTED_BEFORE_EFFECT`, and self-repair can take it.
2. **Propose.** The repair agent is shown the step's intent, the recorded
evidence (the target crop, its label, its place), the live screen, the
halt kind, and for `after` the text of the screen the step expected. It
Expand Down
7 changes: 6 additions & 1 deletion packages/flow/openadapt_flow/benchmark/reliability.py
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,12 @@ def classify_failure(report: Optional[RunReport], error: Optional[str]) -> dict:
return {"category": "none", "root_cause_hint": None}

err = (failing.error or "").lower()
if failing.resolution is None and "all resolution rungs failed" in err:
if failing.resolution is None and (
"all resolution rungs failed" in err
# A closed-loop scroll that never found the target hands the halt to
# that target's step (#1331).
or err.startswith("could not resolve target for step")
):
category = "resolution_failed"
elif "identity check failed" in err:
category = "identity_mismatch"
Expand Down
28 changes: 26 additions & 2 deletions packages/flow/openadapt_flow/repair/self_repair.py
Original file line number Diff line number Diff line change
Expand Up @@ -697,8 +697,7 @@ def run(self, bundle_dir: Path | str, run_dir: Path | str) -> SelfRepairResult:
if report.transaction_outcome not in (None, "HALTED_BEFORE_EFFECT"):
return self._finish(
"not_repairable",
f"the run ended {report.transaction_outcome}; self-repair "
"handles only a halt before any effect",
_not_before_effect(report),
evidence,
report,
run_dir,
Expand Down Expand Up @@ -2152,6 +2151,31 @@ def _recorded_crop(workflow: Workflow, bundle_dir: Path, step: Step) -> Optional
return path.read_bytes() if path.is_file() else None


def _not_before_effect(report: RunReport) -> str:
"""Why a halt that may have left an effect is not repaired, and by whom."""

from openadapt_flow.transaction import steps_requiring_reconciliation

reason = (
f"the run ended {report.transaction_outcome}; self-repair handles only "
"a halt before any effect"
)
unsettled = steps_requiring_reconciliation(report)
if not unsettled:
return reason
reason += f" (effect not settled at {', '.join(unsettled[:8])})"
unreviewed = {r.step_id for r in report.results if r.risk_review_required}
if unreviewed.issuperset(unsettled):
# Only steps nobody classified keep this run from an absence claim.
# A person reviews them once; the rule itself stays fail-closed.
reason += (
"; each acted with a risk nobody reviewed, so a write cannot be "
"ruled out. A person can classify them with `openadapt-flow "
"qualify set-risk`"
)
return reason[:1024]


def _repairable(report: RunReport, workflow: Workflow) -> "tuple[Step, str, Any] | str":
"""``(step, mode, result)`` when self-repair may handle this halt, else why not.

Expand Down
Loading
Loading