Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitleaksignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ eb78575c688f06d7edc04afc69389aad10e1f5fa:packages/flow/tests/fixtures/terminal_v
# branch commit that added them. The lines carry gitleaks:allow from the next commit on.
3f0df2191a649540e79a621ac5406ebe2ff228e2:packages/flow/openadapt_flow/jit/authority.py:generic-api-key:101
3f0df2191a649540e79a621ac5406ebe2ff228e2:packages/flow/openadapt_flow/jit/authority.py:generic-api-key:390
# PR #1354: false positive on the code `Workflow.load(source_path, key=key,
# verify_integrity=True)` in bundle_sealing.py; no secret (the key comes from
# OPENADAPT_BUNDLE_KEY at run time).
a94ffbebccba96d9c2f9ca2016413f10c7526ff1:packages/flow/openadapt_flow/bundle_sealing.py:generic-api-key:282
# PR #1367 (network capture redaction): the first commit's synthetic test values in
# test_network_capture_redact_json.py; the next commit builds them at run time.
52502adf407793c211fb3c0ac5aba37556206d8c:packages/flow/tests/test_network_capture_redact_json.py:generic-api-key:17
Expand Down
1 change: 1 addition & 0 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ improvising or retrying when something is off.
| Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](packages/privacy/README.md), [PRIVACY.md](packages/flow/docs/PRIVACY.md) |
| Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` |
| Each package installs on its own within a size budget, checked in CI (privacy without desktop, capture without plotting) | On main ([#1270](https://github.com/OpenAdaptAI/OpenAdapt/pull/1270), [#1271](https://github.com/OpenAdaptAI/OpenAdapt/pull/1271), [#1279](https://github.com/OpenAdaptAI/OpenAdapt/pull/1279)) | CI | `.github/workflows/packages.yml` |
| Encrypt PHI at rest: every bundle file that can hold patient data under one sealed index (unknown files refused), web recordings from the first frame, and the run directories of sealed bundles; compile, induce, scaffold-verifier, qualify, admit, run, resume and undo read them in memory and refuse without the key | In review | Synthetic (no-plaintext audit-hook test over record, compile and a VERIFIED run; per-frame cost measured) | `openadapt-flow seal`, `record`; [phi_at_rest.md](packages/flow/docs/phi_at_rest.md) |
| Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` |
| Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](packages/flow/docs/ON_PREM.md) |

Expand Down
194 changes: 176 additions & 18 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -416,9 +416,10 @@ shipping in the next release; a pull request number means it is being merged.
review, campaign, canary, rollback), agent-demonstrated fixes for a halted
run, and fleet drift patches shared with no patient data are released.
Self-repair, where an agent proposes a fix and Flow probes it up to the save
without saving, is on main; fixes of up to three new steps are in review.
9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded
save around any agent's write
without saving, is on main, and so are fixes of up to three new steps.
9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task
ledger that keeps a long job's rules, items, questions and checked saves on
the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write
([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run`
to judge any executor's write from the system of record, and Agent Skills and
a CLI for coding agents. Released except where a PR is named.
Expand Down Expand Up @@ -574,6 +575,7 @@ improvising or retrying when something is off.
| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) |
| Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) |
| MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) |
| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) |
| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` |
| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` |
| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` |
Expand All @@ -586,24 +588,118 @@ improvising or retrying when something is off.
| Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/privacy/README.md), [PRIVACY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PRIVACY.md) |
| Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` |
| Each package installs on its own within a size budget, checked in CI (privacy without desktop, capture without plotting) | On main ([#1270](https://github.com/OpenAdaptAI/OpenAdapt/pull/1270), [#1271](https://github.com/OpenAdaptAI/OpenAdapt/pull/1271), [#1279](https://github.com/OpenAdaptAI/OpenAdapt/pull/1279)) | CI | `.github/workflows/packages.yml` |
| Encrypt PHI at rest: every bundle file that can hold patient data under one sealed index (unknown files refused), web recordings from the first frame, and the run directories of sealed bundles; compile, induce, scaffold-verifier, qualify, admit, run, resume and undo read them in memory and refuse without the key | In review | Synthetic (no-plaintext audit-hook test over record, compile and a VERIFIED run; per-frame cost measured) | `openadapt-flow seal`, `record`; [phi_at_rest.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/phi_at_rest.md) |
| Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` |
| Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ON_PREM.md) |

## Every command, by capability

Each `openadapt-flow` command, grouped by the capability it serves. A test
fails when a command is added without being listed here.

- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`.
- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`.
- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`.
- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record).
- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop).
- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`.
- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`.
- **Run on premises:** `service`, `no-outbound`.
- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`.
- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`.
Each `openadapt-flow` command, grouped by the capability it serves. Each
command has one file under
[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a
command, add `<group>/<verb>.md` and run `python scripts/build_command_map.py`.
A test fails when a command has no file or this list is out of date.

<!-- BEGIN generated by scripts/build_command_map.py -->

**Record and build a program:**

- `compile-agent`
- `compile`
- `compose`
- `demo-record`
- `disambiguate`
- `for-each`
- `induce`
- `intake`
- `process`
- `pull`
- `record`
- `route`
- `variants`
- `visualize`

**Check and admit:**

- `admit`
- `api`
- `approver-grant`
- `bind-record-checks`
- `certify`
- `check`
- `jit`
- `known-ids`
- `lint`
- `propose-qualification`
- `qualify`
- `scaffold-verifier`

**Run and answer:**

- `approve`
- `business-decisions`
- `console`
- `deferred`
- `explain`
- `item-approval`
- `replay`
- `resume`
- `run`
- `triage`

**Judge any executor's write:**

- `serve-reward`: Scores agent training episodes by reading the system of record.
- `verify-run`

**Repair and learn from halts:**

- `fleet`
- `repair`
- `teach-branch`
- `teach`
- `undo`: The undo steps a person approves after a stop.

**Give agents the program:**

- `emit-mcp`
- `emit-skill`
- `serve-execute`

**Proof:**

- `attach-seal`
- `countersign`
- `local-seal`
- `seal`

**Run on premises:**

- `no-outbound`
- `service`

**Share safely with OpenAdapt Cloud:**

- `approve-sanitized`
- `connect`
- `connector`
- `login`
- `materialize-approved`
- `push`
- `report-break`
- `report-run`
- `review-sanitized`
- `sanitize`
- `validate-hosted`

**Try it and measure it:**

- `bench`
- `benchmark`
- `demo`
- `tutorial`

<!-- END generated by scripts/build_command_map.py -->

## Measured results

Expand Down Expand Up @@ -1128,6 +1224,30 @@ halts with a named reason when the confirmation is missing
before and after numbers are in
[`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md).

Before a step stops on a look-alike identifier, the replayer takes extra looks
that change nothing
([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)):
the same read on two fresh screenshots, the 2x re-read, the known-ID list when
one is configured, and, where the surface allows it, a select-and-copy of the
identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle
the record on their own, because a look-alike twin can read the same on every
frame and at every size. Only two looks can: the known-ID list, and a copied
identifier that equals a run parameter in the identity band. A look that names
another record stops the run. Each look is recorded on the step's identity
check.

The copy look has limits. It needs a surface where selecting the identifier
and copying it changes nothing, and a clipboard that reaches the machine Flow
runs on (RDP or Citrix clipboard redirection). It drags across only the one
text box that holds the identifier, and the run stops if the screen changes
anywhere else during the look. A canvas with no selectable text has no copy
look. A recording's own identifier is OCR text and can be a misread (the RDP
kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the
recorded record, the copy can stop a run but never confirm one. It confirms
only a record chosen by a run parameter. Governed and qualification runs never
take it. The check just before the click takes no input; it reuses the copy
only when the identifier's pixels are unchanged since the look.

Permissive `replay` may proceed on an unreadable reversible target and reports
that condition. Governed `run` requires an affirmative live verdict for every
identity-required admitted step; mismatch, unreadable, or abstain halts before
Expand Down Expand Up @@ -1156,6 +1276,39 @@ irreversible, and qualification requires its record check. An icon-only click
that only navigates after data entry is therefore over-classified until an
operator reclassifies it.

On a pixel surface an icon-only click carries no text that names its control.
When the recorder can move the pointer without clicking
(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the
pointer on each click target first and keeps that frame. For a click with no
other identity evidence, the compiler stores what changed near the target, such
as the control's highlight or tooltip, as the step's hover evidence
([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)).
At replay, on the pixel rungs, the runtime rests the pointer on the resolved
target, waits the same 0.8 s, and compares. A different response, or a
response where the recording saw none, stops the run before the click. A
missing response, or one that can't be compared (another render with no
readable text), is recorded on the step and changes nothing. A tooltip that
opens later than the wait looks like a different response and stops the run.
Controls that don't answer the pointer, such as most text fields, get no
evidence this way. The browser, remote-display (Citrix and RDP client window)
and FreeRDP backends can hover. The Aardwolf RDP transport and the native
Windows, macOS and Linux backends can't move the pointer without a button
yet, so they take no hover look.

A pixel step is checked on one frame and clicked a moment later. Consequential
identity-gated clicks were already bound to the exact checked frame. Every
other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`):
right before the click, the runtime takes a fresh frame and compares the
target region, the identity region and the hover response with the frame the
step was checked on. If one changed (a dialog opened, the list re-sorted), it
doesn't click. It re-resolves the target and re-runs the identity check on
the fresh frame, with no input, and clicks only when the next frame matches
that one. After two re-observations that still see a change, or when the
target or its identity can't be confirmed again, it refuses the click. Each
change is recorded on the step (`StepResult.check_to_click`). A change smaller
than 4 pixels, or under 24 levels per channel, doesn't count. Structured
targets are bound by element tokens instead and aren't rechecked.

A consequential click must be tied to a record before Flow delivers it. Inside
a table row, the row is that record. An icon-only Save that floats beside a
form (openIMIS's claim Save has no text, accessible name, or tooltip) has no
Expand Down Expand Up @@ -1568,8 +1721,13 @@ host hardening, storage encryption, backups, retention, access control,
monitoring, effect-verifier integration, update validation, and disaster
recovery.

The repository supplies integrity manifests, optional bundle encryption,
destination policy, local egress checks, and audit primitives. A content digest
The repository supplies integrity manifests, optional encryption at rest for
bundles, web recordings and the run directories of sealed bundles, destination
policy, local egress checks, and audit primitives. Encryption at rest does not
cover data in process memory, the target application's own storage, file names
and sizes, desktop recordings, or commands outside the list in
[`phi_at_rest.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/phi_at_rest.md), which refuse sealed input; the key is a
passphrase in the environment, not a KMS. A content digest
proves byte identity, not publisher identity. A local hash chain is tamper
evident against ordinary changes, not tamper proof against an administrator who
can rewrite it. Local approvals record provenance but are not enterprise-IdP
Expand Down
10 changes: 7 additions & 3 deletions packages/flow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,9 +385,13 @@ export OPENADAPT_FLOW_SCRUB=on
The bundle and `report.json` keep literal identifiers, because the identity
check and the audit trail need them ([docs/PRIVACY.md](docs/PRIVACY.md)). With
`OPENADAPT_BUNDLE_KEY` set, `openadapt-flow seal ./bundle --out ./bundle-sealed`
encrypts `workflow.json` and the image crops with AES-256-GCM. Other bundle
files aren't encrypted, so treat every source bundle as PHI
([docs/phi_at_rest.md](docs/phi_at_rest.md)).
encrypts every bundle file that can hold patient data with AES-256-GCM, binds
the whole file set to one sealed index, and refuses a file it doesn't know.
Runs of a sealed bundle write an encrypted run directory, and web recordings
are encrypted from the first frame (`OPENADAPT_RECORDING_KEY`, or the bundle
key). Data in memory, the app's own storage, file names and sizes, and desktop
recordings are not encrypted; the source bundle you sealed is still plaintext,
so treat it as PHI ([docs/phi_at_rest.md](docs/phi_at_rest.md)).

OpenAdapt Cloud at `app.openadapt.ai` is an optional hosted control plane for
browser workflows. Desktop and Citrix runs are self-hosted
Expand Down
Loading
Loading