diff --git a/llms-full.txt b/llms-full.txt index 7f4752fa6..230998709 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -416,9 +416,10 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is on main; fixes of up to three new steps are in review. -9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded - save around any agent's write + without saving, is on main, and so are fixes of up to three new steps. +9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task + ledger that keeps a long job's rules, items, questions and checked saves on + the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. @@ -574,6 +575,7 @@ improvising or retrying when something is off. | Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | | Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | @@ -591,19 +593,112 @@ improvising or retrying when something is off. ## Every command, by capability -Each `openadapt-flow` command, grouped by the capability it serves. A test -fails when a command is added without being listed here. - -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. -- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). -- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. -- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. -- **Run on premises:** `service`, `no-outbound`. -- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`. -- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`. +Each `openadapt-flow` command, grouped by the capability it serves. Each +command has one file under +[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a +command, add `/.md` and run `python scripts/build_command_map.py`. +A test fails when a command has no file or this list is out of date. + + + +**Record and build a program:** + +- `compile-agent` +- `compile` +- `compose` +- `demo-record` +- `disambiguate` +- `for-each` +- `induce` +- `intake` +- `process` +- `pull` +- `record` +- `route` +- `variants` +- `visualize` + +**Check and admit:** + +- `admit` +- `api` +- `approver-grant` +- `bind-record-checks` +- `certify` +- `check` +- `jit` +- `known-ids` +- `lint` +- `propose-qualification` +- `qualify` +- `scaffold-verifier` + +**Run and answer:** + +- `approve` +- `business-decisions` +- `console` +- `deferred` +- `explain` +- `item-approval` +- `replay` +- `resume` +- `run` +- `triage` + +**Judge any executor's write:** + +- `serve-reward`: Scores agent training episodes by reading the system of record. +- `verify-run` + +**Repair and learn from halts:** + +- `fleet` +- `repair` +- `teach-branch` +- `teach` +- `undo`: The undo steps a person approves after a stop. + +**Give agents the program:** + +- `emit-mcp` +- `emit-skill` +- `serve-execute` + +**Proof:** + +- `attach-seal` +- `countersign` +- `local-seal` +- `seal` + +**Run on premises:** + +- `no-outbound` +- `service` + +**Share safely with OpenAdapt Cloud:** + +- `approve-sanitized` +- `connect` +- `connector` +- `login` +- `materialize-approved` +- `push` +- `report-break` +- `report-run` +- `review-sanitized` +- `sanitize` +- `validate-hosted` + +**Try it and measure it:** + +- `bench` +- `benchmark` +- `demo` +- `tutorial` + + ## Measured results @@ -1128,6 +1223,30 @@ halts with a named reason when the confirmation is missing before and after numbers are in [`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md). +Before a step stops on a look-alike identifier, the replayer takes extra looks +that change nothing +([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)): +the same read on two fresh screenshots, the 2x re-read, the known-ID list when +one is configured, and, where the surface allows it, a select-and-copy of the +identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle +the record on their own, because a look-alike twin can read the same on every +frame and at every size. Only two looks can: the known-ID list, and a copied +identifier that equals a run parameter in the identity band. A look that names +another record stops the run. Each look is recorded on the step's identity +check. + +The copy look has limits. It needs a surface where selecting the identifier +and copying it changes nothing, and a clipboard that reaches the machine Flow +runs on (RDP or Citrix clipboard redirection). It drags across only the one +text box that holds the identifier, and the run stops if the screen changes +anywhere else during the look. A canvas with no selectable text has no copy +look. A recording's own identifier is OCR text and can be a misread (the RDP +kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the +recorded record, the copy can stop a run but never confirm one. It confirms +only a record chosen by a run parameter. Governed and qualification runs never +take it. The check just before the click takes no input; it reuses the copy +only when the identifier's pixels are unchanged since the look. + Permissive `replay` may proceed on an unreadable reversible target and reports that condition. Governed `run` requires an affirmative live verdict for every identity-required admitted step; mismatch, unreadable, or abstain halts before @@ -1156,6 +1275,39 @@ irreversible, and qualification requires its record check. An icon-only click that only navigates after data entry is therefore over-classified until an operator reclassifies it. +On a pixel surface an icon-only click carries no text that names its control. +When the recorder can move the pointer without clicking +(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the +pointer on each click target first and keeps that frame. For a click with no +other identity evidence, the compiler stores what changed near the target, such +as the control's highlight or tooltip, as the step's hover evidence +([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)). +At replay, on the pixel rungs, the runtime rests the pointer on the resolved +target, waits the same 0.8 s, and compares. A different response, or a +response where the recording saw none, stops the run before the click. A +missing response, or one that can't be compared (another render with no +readable text), is recorded on the step and changes nothing. A tooltip that +opens later than the wait looks like a different response and stops the run. +Controls that don't answer the pointer, such as most text fields, get no +evidence this way. The browser, remote-display (Citrix and RDP client window) +and FreeRDP backends can hover. The Aardwolf RDP transport and the native +Windows, macOS and Linux backends can't move the pointer without a button +yet, so they take no hover look. + +A pixel step is checked on one frame and clicked a moment later. Consequential +identity-gated clicks were already bound to the exact checked frame. Every +other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`): +right before the click, the runtime takes a fresh frame and compares the +target region, the identity region and the hover response with the frame the +step was checked on. If one changed (a dialog opened, the list re-sorted), it +doesn't click. It re-resolves the target and re-runs the identity check on +the fresh frame, with no input, and clicks only when the next frame matches +that one. After two re-observations that still see a change, or when the +target or its identity can't be confirmed again, it refuses the click. Each +change is recorded on the step (`StepResult.check_to_click`). A change smaller +than 4 pixels, or under 24 levels per channel, doesn't count. Structured +targets are bound by element tokens instead and aren't rechecked. + A consequential click must be tied to a record before Flow delivers it. Inside a table row, the row is that record. An icon-only Save that floats beside a form (openIMIS's claim Save has no text, accessible name, or tooltip) has no diff --git a/packages/flow/claims/server-response-evidence.yaml b/packages/flow/claims/server-response-evidence.yaml new file mode 100644 index 000000000..88b12c782 --- /dev/null +++ b/packages/flow/claims/server-response-evidence.yaml @@ -0,0 +1,25 @@ +id: server-response-evidence +claim: >- + In a browser, the server's answer to a save's own write request is + kept on the step and counted on the receipt at check level + same_screen, as method, status and media type only. It comes from the + same transaction as the save, so on its own it never confirms an effect + or makes a run VERIFIED. +surfaces: [docs] +tier: supported +evidence: + - path: tests/test_server_responses.py + proves: >- + Only same-origin fetch, XHR and form writes sent after the mark are + kept, never beacons, other origins, requests already in flight or + service-worker answers, and never a URL, path or body; a real + Chromium save produces one; a 201 answer with an unconfirmed + read-back still ends RECONCILIATION_REQUIRED and not billable; a + redirect counts as not an error; the receipt counts answers without + changing its evidence tier; results without answers serialize as + before. +caveats: + - >- + Only the browser backend sees server answers. A save that writes to + another origin (a separate API host) leaves none. An answer that + arrives after the screen postconditions is not kept. diff --git a/packages/flow/docs/APPROVAL_AND_VERIFICATION.md b/packages/flow/docs/APPROVAL_AND_VERIFICATION.md index a6dae48ca..641a6485a 100644 --- a/packages/flow/docs/APPROVAL_AND_VERIFICATION.md +++ b/packages/flow/docs/APPROVAL_AND_VERIFICATION.md @@ -196,6 +196,13 @@ older scales keep their published numbers and map onto it: | 1 `same_screen` | Only the acting screen shows the value | 4 `IMMEDIATE_SCREEN` | 0 `VISUAL` | `screen_readback` | No | | 0 `unchecked` | No read, unreachable or stale store, no tier | (no confirmed tier) | (no Seal) | `unchecked` | No | +The server's own answer to a save, when the browser sees it, is kept too +(`StepResult.server_responses`: method, status and media type, never a URL, +a body or a digest of one; counted on the receipt as `server_responses`). It ranks `same_screen`: +it comes from the same transaction as the save, so on its own it never +confirms a save, never makes a run `VERIFIED`, and never makes a save clean. +A 500 behind a "Saved" banner is still worth seeing. + `VerificationTier` counts the other way (lower is stronger), so compare checks by `.check_level`, not by the tier number. A Seal at `VISUAL` reads as `same_screen`, because the Seal does not record whether the screen was diff --git a/packages/flow/docs/EFFECT_KIT.md b/packages/flow/docs/EFFECT_KIT.md index aa76328c2..ccba629c7 100644 --- a/packages/flow/docs/EFFECT_KIT.md +++ b/packages/flow/docs/EFFECT_KIT.md @@ -897,6 +897,30 @@ Citrix session. - Citrix is a separate Workspace window named in `session`. Opening a new Citrix session for each check is the operator's launcher, not Flow's. +## The server's own answer (kept, labelled `same_screen`) + +In a browser, a save usually sends a write request (POST, PUT, PATCH, +DELETE) and the server answers it. Flow keeps that answer on the save step +(`StepResult.server_responses`) and counts it on the receipt +(`server_responses`: observed, not errors, `check_level: same_screen`). It +keeps the method, the status and the media type. It never keeps a URL, a path, +a header, a body, or a digest of any of them (a digest of a short path or body +can be guessed), and it never reads a response body, so a streaming answer +can't hold up the run. + +Only the page's own write requests count: `fetch`, XHR and form submissions +to the page's origin, sent after the save began. A beacon, a request to +another origin (analytics, a widget), a request already in flight, and an +answer from a service worker are left out. "Not an error" means a status +below 400, because a form save often answers with a redirect. + +The answer comes from the same transaction as the save. A server can say +`201 Created` and store the wrong value, or store it under another record, or +roll it back afterwards. So the answer is evidence at check level +`same_screen`, never a check: on its own it never confirms an effect and +never makes a run `VERIFIED`. The configured verifier still decides. Only the +browser backend sees answers; desktop, RDP and Citrix runs have none. + ## Checked later (`deferred_report`) Some saves can only be confirmed by a report that arrives after the run: a diff --git a/packages/flow/docs/VERIFICATION.md b/packages/flow/docs/VERIFICATION.md index 1305f0616..2ff8922f4 100644 --- a/packages/flow/docs/VERIFICATION.md +++ b/packages/flow/docs/VERIFICATION.md @@ -27,6 +27,7 @@ - `qualified-remote-decision-v2-supported` - `record-check-from-app-reads` - `reviewed-judgment-cases-supported` +- `server-response-evidence` - `substrate-runtime-validation-supported` - `typed-business-decisions-supported` - `ui-readback-separate-session` @@ -274,6 +275,21 @@ - An automatic-rule case names only a reviewed rule identifier and finite option. Flow does not infer or generate executable policy from the case; the rule must be authored and qualified through the normal program path. - A retained human-node case preserves runtime human authority. A more-evidence-required case refuses certification instead of guessing. +### `server-response-evidence` — supported — bound to required CI pass evidence + +> In a browser, the server's answer to a save's own write request is kept on the step and counted on the receipt at check level same_screen, as method, status and media type only. It comes from the same transaction as the save, so on its own it never confirms an effect or makes a run VERIFIED. + +- Surfaces: docs +- Strongest evidence strength: **supported** (tier is `supported`) + +| Backing evidence | Kind | Gating / CI stage | Strength | Proves | +|---|---|---|---|---| +| `tests/test_server_responses.py` | test | ci (required PR gate (test)) | supported | Only same-origin fetch, XHR and form writes sent after the mark are kept, never beacons, other origins, requests already in flight or service-worker answers, and never a URL, path or body; a real Chromium save produces one; a 201 answer with an unconfirmed read-back still ends RECONCILIATION_REQUIRED and not billable; a redirect counts as not an error; the receipt counts answers without changing its evidence tier; results without answers serialize as before. | + +**Caveats (honest limits):** + +- Only the browser backend sees server answers. A save that writes to another origin (a separate API host) leaves none. An answer that arrives after the screen postconditions is not kept. + ### `substrate-runtime-validation-supported` — supported — bound to required CI pass evidence > Runtime-validation v3 binds Standard and Regulated hosted activation to the exact governed authorization template reproduced from the sealed qualification. It also binds the successful run's resolved target kind (`web|windows|macos|linux|rdp|citrix`), artifact, compiler, parameter schema, strict lint, policy, risk, report, and opaque environment. Web retains its exact HTTPS/host boundary. Native and remote execution emits no app, window, host, readiness, or backend-hint values to Cloud. Local verification retains signed v1 and v2 compatibility. diff --git a/packages/flow/docs/verification.json b/packages/flow/docs/verification.json index c736da2e5..fb1320194 100644 --- a/packages/flow/docs/verification.json +++ b/packages/flow/docs/verification.json @@ -682,6 +682,34 @@ ], "errors": [] }, + { + "id": "server-response-evidence", + "claim": "In a browser, the server's answer to a save's own write request is kept on the step and counted on the receipt at check level same_screen, as method, status and media type only. It comes from the same transaction as the save, so on its own it never confirms an effect or makes a run VERIFIED.", + "tier": "supported", + "reproducibility": null, + "surfaces": [ + "docs" + ], + "strongest_evidence": "supported", + "caveats": [ + "Only the browser backend sees server answers. A save that writes to another origin (a separate API host) leaves none. An answer that arrives after the screen postconditions is not kept." + ], + "evidence": [ + { + "path": "tests/test_server_responses.py", + "kind": "test", + "exists": true, + "strength": "supported", + "gating": "ci (required PR gate (test))", + "node": null, + "node_found": null, + "ci_job": "test", + "junit_status": null, + "proves": "Only same-origin fetch, XHR and form writes sent after the mark are kept, never beacons, other origins, requests already in flight or service-worker answers, and never a URL, path or body; a real Chromium save produces one; a 201 answer with an unconfirmed read-back still ends RECONCILIATION_REQUIRED and not billable; a redirect counts as not an error; the receipt counts answers without changing its evidence tier; results without answers serialize as before." + } + ], + "errors": [] + }, { "id": "substrate-runtime-validation-supported", "claim": "Runtime-validation v3 binds Standard and Regulated hosted activation to the exact governed authorization template reproduced from the sealed qualification. It also binds the successful run's resolved target kind (`web|windows|macos|linux|rdp|citrix`), artifact, compiler, parameter schema, strict lint, policy, risk, report, and opaque environment. Web retains its exact HTTPS/host boundary. Native and remote execution emits no app, window, host, readiness, or backend-hint values to Cloud. Local verification retains signed v1 and v2 compatibility.", diff --git a/packages/flow/openadapt_flow/backends/_server_responses.py b/packages/flow/openadapt_flow/backends/_server_responses.py new file mode 100644 index 000000000..f4e6ad66f --- /dev/null +++ b/packages/flow/openadapt_flow/backends/_server_responses.py @@ -0,0 +1,166 @@ +"""The server's answers to a save's write requests, as the browser sees them. + +When a person or a program presses Save in a web application, the browser +usually sends a write request (POST, PUT, PATCH, DELETE) and the server +answers it. That answer is worth keeping: a 500 behind a green "Saved" +banner is a clear sign something went wrong. It is not proof that the record +is right, because it comes from the same transaction as the save. The +replayer keeps it on the step as evidence at check level ``same_screen``, +and only a read-back decides whether the save is confirmed. + +:class:`ServerResponseLog` listens to one Playwright page. It keeps answers +to write requests that: + +- the page's own code sent (``fetch``, XHR, or a form submission), not a + beacon, an image or a script; +- went to the page's own origin; +- were sent after the save began (requests already in flight are ignored); +- came from the network, not from a service worker (an offline-first app's + worker can answer a write that never reached the server). + +It keeps the method, the status and the media type, and whether the request +went to the page's origin. It never keeps a URL, a path, a header, a body or +a digest of any of them: a digest of a short path or body can be guessed. It +never reads a response body, so a streaming answer can't hold up the run. +""" + +from __future__ import annotations + +import threading +from collections import deque +from typing import Any, Optional +from urllib.parse import urlsplit + +from openadapt_flow.ir import ServerResponseEvidence + +WRITE_METHODS = frozenset({"POST", "PUT", "PATCH", "DELETE"}) +#: Requests the page's own code makes: fetch, XHR, and form submissions. +_PAGE_REQUESTS = frozenset({"fetch", "xhr", "document"}) +#: Answers kept in the log; older ones drop off. +_MAX_LOG = 64 + + +def _origin(url: str) -> str: + parts = urlsplit(url) + return f"{parts.scheme}://{parts.netloc}".lower() + + +class ServerResponseLog: + """A bounded log of answers to the page's own write requests.""" + + def __init__(self, page: Any) -> None: + self._page = page + self._lock = threading.Lock() + #: Sequence number of each tracked write request, by request object. + self._issued: dict[int, int] = {} + self._seq = 0 + self._log: deque[tuple[int, ServerResponseEvidence]] = deque(maxlen=_MAX_LOG) + #: Listeners are added at the first :meth:`mark` (the first save), not + #: before: nothing earlier is kept, and a page that never saves gets + #: no extra listeners. + self.armed = False + + def _arm(self) -> None: + if self.armed: + return + try: + self._page.on("request", self._on_request) + self._page.on("response", self._on_response) + self.armed = True + except Exception: # noqa: BLE001 - evidence only + self.armed = False + + def close(self) -> None: + """Remove the listeners (the backend is done with the page).""" + + if not self.armed: + return + for event, listener in ( + ("request", self._on_request), + ("response", self._on_response), + ): + try: + self._page.remove_listener(event, listener) + except Exception: # noqa: BLE001 - best-effort teardown + pass + self.armed = False + + def _on_request(self, request: Any) -> None: + try: + method = str(request.method).upper() + kind = str(request.resource_type) + except Exception: # noqa: BLE001 + return + if method not in WRITE_METHODS or kind not in _PAGE_REQUESTS: + return + with self._lock: + self._seq += 1 + self._issued[id(request)] = self._seq + if len(self._issued) > 4 * _MAX_LOG: + # Requests that never got an answer don't pile up. + oldest = sorted(self._issued.items(), key=lambda kv: kv[1]) + for key, _ in oldest[: len(oldest) - 2 * _MAX_LOG]: + self._issued.pop(key, None) + + def _on_response(self, response: Any) -> None: + try: + request = response.request + except Exception: # noqa: BLE001 + return + with self._lock: + seq = self._issued.pop(id(request), None) + if seq is None: + return + evidence = _evidence(self._page, response) + if evidence is None: + return + with self._lock: + self._log.append((seq, evidence)) + + def mark(self) -> int: + """Requests sent after this mark belong to what comes next.""" + + self._arm() + with self._lock: + return self._seq + + def since(self, mark: int) -> list[ServerResponseEvidence]: + """Answers to write requests sent after ``mark`` (oldest first).""" + + with self._lock: + return [evidence for seq, evidence in self._log if seq > mark] + + +def _evidence(page: Any, response: Any) -> Optional[ServerResponseEvidence]: + try: + if getattr(response, "from_service_worker", False): + return None + method = str(response.request.method).upper() + status = int(response.status) + url = str(response.url) + except Exception: # noqa: BLE001 - a response we can't describe is skipped + return None + if method not in WRITE_METHODS or not 100 <= status <= 599: + return None + try: + page_origin = _origin(str(page.url)) + except Exception: # noqa: BLE001 + return None + if _origin(url) != page_origin: + # Another origin: analytics, a third-party widget. Not the save. + return None + content_type: Optional[str] = None + try: + raw = (response.headers or {}).get("content-type") + if raw: + content_type = str(raw).split(";", 1)[0].strip().lower()[:100] or None + except Exception: # noqa: BLE001 + content_type = None + return ServerResponseEvidence( + method=method, # type: ignore[arg-type] + status=status, + content_type=content_type, + ) + + +__all__ = ["ServerResponseLog", "WRITE_METHODS"] diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index d30d4598f..c2f6d9398 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -22,6 +22,7 @@ from urllib.parse import urlsplit from openadapt_flow.backends._page_quiescence import PageQuiescence +from openadapt_flow.backends._server_responses import ServerResponseLog from openadapt_flow.runtime.step_timing import timed_span if TYPE_CHECKING: # pragma: no cover @@ -1173,6 +1174,18 @@ def __init__( # openadapt_flow.backends._page_quiescence). Armed at construction: # network requests issued before this point are not tracked. self._quiescence = PageQuiescence(page) + # The server's answers to write requests, kept as save evidence (see + # openadapt_flow.backends._server_responses). Never a check. + self._server_responses = ServerResponseLog(page) + + def server_response_mark(self) -> int: + """Where the log of server answers to write requests stands now.""" + return self._server_responses.mark() + + def server_responses_since(self, mark: int) -> list[Any]: + """Server answers to the page's own write requests sent after + ``mark``; never blocks on a response body.""" + return self._server_responses.since(mark) @property def viewport(self) -> tuple[int, int]: diff --git a/packages/flow/openadapt_flow/ir.py b/packages/flow/openadapt_flow/ir.py index dca06b1fd..70bee42fb 100644 --- a/packages/flow/openadapt_flow/ir.py +++ b/packages/flow/openadapt_flow/ir.py @@ -3905,6 +3905,31 @@ class EffectFieldCheck(BaseModel): verification_tier: Optional[int] = None +class ServerResponseEvidence(BaseModel): + """The application server's answer to a save request, as the browser saw it. + + Kept on the step as evidence at check level ``same_screen``: it comes from + the same transaction as the save, so on its own it never confirms an + effect and never makes a run VERIFIED. It holds the method, the status and + the media type only: no URL, path, body, or digest of one (a digest of a + short path or body can be guessed). + """ + + model_config = ConfigDict(extra="forbid") + + method: Literal["POST", "PUT", "PATCH", "DELETE"] + status: int = Field(ge=100, le=599) + #: The response's media type (``application/json``), without parameters. + content_type: Optional[str] = Field(default=None, max_length=100) + #: Where this evidence ranks (:mod:`openadapt_flow.check_level`). + check_level: Literal["same_screen"] = "same_screen" + + @property + def ok(self) -> bool: + """Not an error: 2xx, or a 3xx redirect after a form post.""" + return self.status < 400 + + class CheckToClickRecheck(BaseModel): """The screen changed between a pixel step's check and its click. @@ -3973,6 +3998,10 @@ class StepResult(BaseModel): safety_halt: bool = False effect_results: list[str] = Field(default_factory=list) effect_evidence: list[EffectVerificationEvidence] = Field(default_factory=list) + #: The server's answers to this step's write requests, when the browser + #: saw them. Evidence at check level ``same_screen``; never a check on its + #: own. Empty (and omitted when serialized) on every other backend. + server_responses: list[ServerResponseEvidence] = Field(default_factory=list) #: Field-by-field read-back comparisons; omitted from the report when empty. effect_field_checks: list[EffectFieldCheck] = Field(default_factory=list) #: Machine-readable terminal category. ``report.halt`` remains useful @@ -4079,6 +4108,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: data.pop("timings", None) if isinstance(data, dict) and not self.effect_field_checks: data.pop("effect_field_checks", None) + if isinstance(data, dict) and not self.server_responses: + data.pop("server_responses", None) if isinstance(data, dict) and not self.check_to_click: data.pop("check_to_click", None) return data diff --git a/packages/flow/openadapt_flow/receipt.py b/packages/flow/openadapt_flow/receipt.py index 0070ff49d..0db3bdbbb 100644 --- a/packages/flow/openadapt_flow/receipt.py +++ b/packages/flow/openadapt_flow/receipt.py @@ -130,6 +130,28 @@ _HOUR_UTC_RE = re.compile(r"^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:00:00Z$") +class ReceiptServerResponses(BaseModel): + """Counts of the server's answers to the run's save requests. + + Supporting evidence at check level ``same_screen``: the answers come from + the same transaction as the saves, so they never stand in for a read-back. + """ + + model_config = ConfigDict(extra="forbid") + + observed: ReceiptCount + #: Answers that were not errors (status below 400; a form save often + #: answers with a redirect). + ok: ReceiptCount + check_level: Literal["same_screen"] = "same_screen" + + @model_validator(mode="after") + def _ok_within_observed(self) -> "ReceiptServerResponses": + if self.ok > self.observed: + raise ValueError("server_responses.ok exceeds observed") + return self + + class ReceiptError(ValueError): """A receipt could not be built or written.""" @@ -196,6 +218,9 @@ class RunReceipt(BaseModel): #: :attr:`effect_tier_reached`. Optional so receipts written before it #: existed still validate (``exclude_none`` keeps their digest). effect_assurance: Optional[ReceiptAssurance] = None + #: The server's answers to the save requests, when the browser saw them. + #: Omitted when there were none, so older receipts keep their bytes. + server_responses: Optional[ReceiptServerResponses] = None authorization_required: ReceiptCount authorization_confirmed: ReceiptCount identity_required: ReceiptCount @@ -307,6 +332,8 @@ def _drop_unset_assurance(self, handler: Any) -> dict[str, Any]: data: dict[str, Any] = handler(self) if self.effect_assurance is None: data.pop("effect_assurance", None) + if self.server_responses is None: + data.pop("server_responses", None) if self.standard_change is None: data.pop("standard_change", None) if self.variant is None: @@ -908,6 +935,17 @@ def _build_receipt( bundle_digest=bundle_digest, generated_at=_hour_utc(getattr(report, "started_at", None)), ) + answers = [ + response + for result in report.results + for response in getattr(result, "server_responses", None) or [] + ] + if answers: + payload["server_responses"] = { + "observed": len(answers), + "ok": sum(1 for answer in answers if answer.ok), + "check_level": "same_screen", + } from openadapt_flow.repair.standard_change import receipt_fields standard_change = receipt_fields(report) @@ -1058,6 +1096,18 @@ def _receipt_rows(receipt: RunReceipt) -> list[tuple[str, str]]: if receipt.effect_assurance else [] ), + *( + [ + ( + "server answers", + f"{receipt.server_responses.ok} of " + f"{receipt.server_responses.observed} ok (same transaction; " + "not a check)", + ) + ] + if receipt.server_responses + else [] + ), ( "identity-armed", f"{receipt.identity_armed} of {receipt.identity_applicable}", diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index a6798ee70..5df20b3db 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -669,6 +669,10 @@ def better_identity_read(first: Any, retry: Any) -> Any: return first +#: Server answers kept per step (a save makes one or a few write requests). +_MAX_SERVER_RESPONSES = 64 + + @dataclass(frozen=True) class _WebFormApprovalView: """An API step's prepared request, as a person is asked to approve it.""" @@ -7478,6 +7482,9 @@ def _run_step_on_active_surface( # System-of-record pre-state, snapshotted just before the action when # the step declares effects (see the block guarding self._act below). effect_pre_state: Optional[EffectState] = None + # Where the backend's log of server answers stood before the save, so + # the answers to this step's write requests can be kept as evidence. + server_mark: Optional[int] = None # The step's effect contracts with every ValueExpr bound to THIS run's # params (P0-3); resolved BEFORE the pre-state snapshot so verification # targets the record this run wrote, not the demonstration's. @@ -8003,6 +8010,8 @@ def _run_step_on_active_surface( else "safety_halt" ) break + if server_mark is None and effect_pre_state is not None: + server_mark = self._server_response_mark() try: self._active_delivery_resolution = resolution self._active_delivery_region = matched_region @@ -8357,6 +8366,11 @@ def _run_step_on_active_surface( # save, a phantom optimistic-UI success, a duplicate write, or a # lost update -- the EffectVerifier reads the REAL record and HALTs # on any non-CONFIRMED verdict (docs/design/EFFECT_VERIFIER.md). + if server_mark is not None: + # The server's own answer to the save: kept as evidence at a + # lower check level. It comes from the same transaction, so it + # never confirms an effect; the verifier below still decides. + self._collect_server_responses(result, server_mark) if ( error is None and not delivery_uncertain @@ -9638,6 +9652,28 @@ def _auto_readback_verifier(self) -> Any: self._readback_verifier = verifier return verifier + def _server_response_mark(self) -> Optional[int]: + """Where the backend's server-answer log stands, or None when the + backend doesn't see server answers (every non-browser backend).""" + + mark = getattr(self.backend, "server_response_mark", None) + if not callable(mark): + return None + try: + return int(mark()) + except Exception: # noqa: BLE001 - evidence only, never a failure + return None + + def _collect_server_responses(self, result: StepResult, mark: int) -> None: + since = getattr(self.backend, "server_responses_since", None) + if not callable(since): + return + try: + responses = list(since(mark)) + except Exception: # noqa: BLE001 - evidence only, never a failure + return + result.server_responses = responses[:_MAX_SERVER_RESPONSES] + @timed_span("readback") def _verify_effects( self, diff --git a/packages/flow/public-artifacts.json b/packages/flow/public-artifacts.json index 84e0f1e47..ebc9c435f 100644 --- a/packages/flow/public-artifacts.json +++ b/packages/flow/public-artifacts.json @@ -551,6 +551,7 @@ "claims/rdp-validating.yaml", "claims/record-check-from-app-reads.yaml", "claims/reviewed-judgment-cases-supported.yaml", + "claims/server-response-evidence.yaml", "claims/substrate-runtime-validation-supported.yaml", "claims/typed-business-decisions-supported.yaml", "claims/ui-readback-separate-session.yaml", diff --git a/packages/flow/tests/test_run_receipt.py b/packages/flow/tests/test_run_receipt.py index 4601e1d5e..a847d0b6e 100644 --- a/packages/flow/tests/test_run_receipt.py +++ b/packages/flow/tests/test_run_receipt.py @@ -80,6 +80,9 @@ "evidence_classes", "effect_tier_reached", "effect_assurance", + # Counts of the server's answers to save requests (same transaction; never + # a check). Counts and a fixed label only. + "server_responses", "authorization_required", "authorization_confirmed", "identity_required", diff --git a/packages/flow/tests/test_server_responses.py b/packages/flow/tests/test_server_responses.py new file mode 100644 index 000000000..05e6a6814 --- /dev/null +++ b/packages/flow/tests/test_server_responses.py @@ -0,0 +1,309 @@ +"""The server's own answer to a save, kept as evidence and labelled honestly. + +When the browser sees the server answer a save request, the answer is kept on +the step and counted on the receipt at check level ``same_screen``. It comes +from the same transaction as the save, so on its own it never confirms an +effect and never makes a run VERIFIED. No body, URL or value is kept. +""" + +from __future__ import annotations + +import http.server +import threading +from pathlib import Path +from typing import Any, Iterator + +import pytest + +from openadapt_flow.backends._server_responses import ServerResponseLog +from openadapt_flow.ir import ( + ActionKind, + Postcondition, + PostconditionKind, + ServerResponseEvidence, + Step, + Workflow, +) +from openadapt_flow.receipt import build_receipt +from openadapt_flow.runtime.effects import ( + Effect, + EffectKind, + EffectState, + EffectVerdict, + Verdict, +) +from openadapt_flow.runtime.replayer import Replayer +from tests.test_replayer import FakeBackend, FakeVision, Match + +BODY = b'{"id": "enc-7", "status": "saved"}' + + +# -- the log, on fakes ------------------------------------------------------------- + + +class _Request: + def __init__(self, method: str, kind: str = "fetch") -> None: + self.method = method + self.resource_type = kind + + +class _Response: + def __init__( + self, request: _Request, status: int, url: str, *, worker: bool = False + ) -> None: + self.request = request + self.status = status + self.url = url + self.headers = {"content-type": "application/json; charset=utf-8"} + self.from_service_worker = worker + + def body(self) -> bytes: # pragma: no cover - must never be called + raise AssertionError("the log must not read response bodies") + + +class _Page: + url = "https://emr.test/patients/MRN-1/encounters" + + def __init__(self) -> None: + self.handlers: dict[str, Any] = {} + + def on(self, event: str, handler: Any) -> None: + self.handlers[event] = handler + + +def _send(page: _Page, request: _Request, status: int, url: str, **kw: Any) -> None: + page.handlers["request"](request) + page.handlers["response"](_Response(request, status, url, **kw)) + + +def test_only_the_saves_own_writes_are_kept_without_values() -> None: + page = _Page() + log = ServerResponseLog(page) + log.mark() # an earlier save armed the log + in_flight = _Request("POST") + page.handlers["request"](in_flight) # an autosave sent before this save + mark = log.mark() + page.handlers["response"](_Response(in_flight, 200, "https://emr.test/autosave")) + _send(page, _Request("POST"), 201, "https://emr.test/api/encounters?p=MRN-1") + _send(page, _Request("GET"), 200, "https://emr.test/api/encounters") + _send(page, _Request("POST", "ping"), 204, "https://emr.test/beacon") + _send(page, _Request("POST"), 200, "https://analytics.example/collect") + _send(page, _Request("POST"), 200, "https://emr.test/queue", worker=True) + [answer] = log.since(mark) + assert answer.method == "POST" and answer.status == 201 and answer.ok + assert answer.content_type == "application/json" + assert answer.check_level == "same_screen" + dumped = answer.model_dump_json() + for leak in ("MRN-1", "emr.test", "encounters", "p="): + assert leak not in dumped + + +def test_a_form_redirect_is_not_an_error_but_a_500_is() -> None: + page = _Page() + log = ServerResponseLog(page) + log.mark() + _send(page, _Request("POST", "document"), 302, "https://emr.test/save") + _send(page, _Request("PUT"), 500, "https://emr.test/api/x") + redirect, failure = log.since(0) + assert redirect.ok and not failure.ok + + +# -- the replayer -------------------------------------------------------------------- + + +class AnsweringBackend(FakeBackend): + """A fake backend whose save gets a server answer, as a browser would see.""" + + def __init__(self, status: int = 201) -> None: + super().__init__() + self.status = status + self.answers: list[ServerResponseEvidence] = [] + + def server_response_mark(self) -> int: + return len(self.answers) + + def server_responses_since(self, mark: int) -> list[ServerResponseEvidence]: + return self.answers[mark:] + + def press(self, key: str) -> None: + super().press(key) + self.answers.append( + ServerResponseEvidence( + method="POST", status=self.status, content_type="application/json" + ) + ) + + +class _Verifier: + """A verifier that can't read the record: the save is unconfirmed.""" + + substrate = "rest" + verification_tier = 1 + + def capture_pre_state(self, context: Any = None) -> EffectState: + return EffectState(substrate="rest", reachable=True, records=[]) + + def verify(self, effect: Effect, before: EffectState, context: Any = None): + return EffectVerdict( + verdict=Verdict.INDETERMINATE, + kind=effect.kind, + substrate="rest", + reason="record store unreachable", + unavailable=True, + ) + + +def _workflow() -> Workflow: + return Workflow( + name="save", + steps=[ + Step( + id="save", + intent="save encounter", + action=ActionKind.KEY, + key="Enter", + risk="irreversible", + expect=[ + Postcondition( + kind=PostconditionKind.TEXT_PRESENT, text="Saved", timeout_s=0.2 + ) + ], + effects=[ + Effect( + kind=EffectKind.RECORD_WRITTEN, + match={"patient_id": "p1"}, + expected_count=1, + timeout_s=0.0, + ) + ], + ) + ], + ) + + +def test_a_201_from_the_server_never_makes_the_run_verified(tmp_path: Path) -> None: + vision = FakeVision() + vision.text_results = { + "Saved": Match(point=(50, 10), region=(30, 5, 40, 10), confidence=0.9) + } + bundle = tmp_path / "bundle" + (bundle / "templates").mkdir(parents=True) + backend = AnsweringBackend(201) + report = Replayer( + backend, vision=vision, effect_verifier=_Verifier(), poll_interval_s=0.01 + ).run(_workflow(), bundle_dir=bundle, run_dir=tmp_path / "run") + [result] = report.results + assert [a.status for a in result.server_responses] == [201] + assert result.server_responses[0].check_level == "same_screen" + # The server said "created"; the record read couldn't confirm it. + assert report.execution_outcome != "VERIFIED" + assert report.transaction_outcome == "RECONCILIATION_REQUIRED" + assert report.transaction_billable is False + saved = (tmp_path / "run" / "report.json").read_text() + assert '"server_responses"' in saved + + +def test_steps_without_answers_serialize_as_before() -> None: + from openadapt_flow.ir import StepResult + + assert ( + "server_responses" + not in StepResult(step_id="a", intent="b", ok=True).model_dump_json() + ) + + +def test_the_receipt_counts_answers_as_supporting_evidence() -> None: + from tests.test_run_receipt import _report + + report = _report() + plain = build_receipt(report) + assert "server_responses" not in plain.model_dump() + report.results[-1].server_responses = [ + ServerResponseEvidence(method="POST", status=201), + ServerResponseEvidence(method="POST", status=500), + ] + receipt = build_receipt(report) + assert receipt.server_responses is not None + assert (receipt.server_responses.observed, receipt.server_responses.ok) == (2, 1) + assert receipt.server_responses.check_level == "same_screen" + # Evidence strength still comes from the read-back alone. + assert receipt.effect_tier_reached == plain.effect_tier_reached + + +# -- a real browser ------------------------------------------------------------------- + + +@pytest.fixture() +def save_server() -> Iterator[str]: + page = ( + b"" + ) + + class Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, *args: object) -> None: + return None + + def do_GET(self) -> None: # noqa: N802 + self.send_response(200) + self.send_header("Content-Type", "text/html") + self.end_headers() + self.wfile.write(page) + + def do_POST(self) -> None: # noqa: N802 + length = int(self.headers.get("Content-Length") or 0) + self.rfile.read(length) + self.send_response(201) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(BODY))) + self.end_headers() + self.wfile.write(BODY) + + server = http.server.HTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield f"http://127.0.0.1:{server.server_address[1]}/" + finally: + server.shutdown() + server.server_close() + + +def test_the_browser_backend_keeps_the_answer_to_a_real_save(save_server: str) -> None: + sync_api = pytest.importorskip("playwright.sync_api") + from openadapt_flow.backends.playwright_backend import PlaywrightBackend + + with sync_api.sync_playwright() as pw: + try: + browser = pw.chromium.launch(headless=True) + except Exception as exc: # pragma: no cover - env specific + pytest.skip(f"no chromium: {exc}") + try: + page = browser.new_page() + backend = PlaywrightBackend(page) + page.goto(save_server) + mark = backend.server_response_mark() + page.click("#save") + page.wait_for_function("document.body.innerText.includes('Saved')") + [answer] = backend.server_responses_since(mark) + finally: + browser.close() + assert answer.method == "POST" and answer.status == 201 + assert answer.content_type == "application/json" + assert "MRN-9" not in answer.model_dump_json() + + +def test_the_page_gets_no_listeners_until_a_save_and_they_can_be_removed() -> None: + class Page(_Page): + def remove_listener(self, event: str, handler: Any) -> None: + if self.handlers.get(event) == handler: + del self.handlers[event] + + page = Page() + log = ServerResponseLog(page) + assert page.handlers == {} # nothing until the first save marks the log + log.mark() + assert set(page.handlers) == {"request", "response"} + log.close() + assert page.handlers == {}