diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c42ee9af9..0b6a65492 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -254,6 +254,7 @@ A test fails when a command has no file or this list is out of date. - `for-each` - `induce` - `intake` +- `narration` - `process` - `pull` - `record` diff --git a/llms-full.txt b/llms-full.txt index 7f4752fa6..46de32869 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -416,9 +416,10 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is on main; fixes of up to three new steps are in review. -9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded - save around any agent's write + without saving, is on main, and so are fixes of up to three new steps. +9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task + ledger that keeps a long job's rules, items, questions and checked saves on + the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. @@ -574,6 +575,7 @@ improvising or retrying when something is off. | Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | | Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | @@ -591,19 +593,113 @@ improvising or retrying when something is off. ## Every command, by capability -Each `openadapt-flow` command, grouped by the capability it serves. A test -fails when a command is added without being listed here. - -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. -- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). -- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. -- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. -- **Run on premises:** `service`, `no-outbound`. -- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`. -- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`. +Each `openadapt-flow` command, grouped by the capability it serves. Each +command has one file under +[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a +command, add `/.md` and run `python scripts/build_command_map.py`. +A test fails when a command has no file or this list is out of date. + + + +**Record and build a program:** + +- `compile-agent` +- `compile` +- `compose` +- `demo-record` +- `disambiguate` +- `for-each` +- `induce` +- `intake` +- `narration` +- `process` +- `pull` +- `record` +- `route` +- `variants` +- `visualize` + +**Check and admit:** + +- `admit` +- `api` +- `approver-grant` +- `bind-record-checks` +- `certify` +- `check` +- `jit` +- `known-ids` +- `lint` +- `propose-qualification` +- `qualify` +- `scaffold-verifier` + +**Run and answer:** + +- `approve` +- `business-decisions` +- `console` +- `deferred` +- `explain` +- `item-approval` +- `replay` +- `resume` +- `run` +- `triage` + +**Judge any executor's write:** + +- `serve-reward`: Scores agent training episodes by reading the system of record. +- `verify-run` + +**Repair and learn from halts:** + +- `fleet` +- `repair` +- `teach-branch` +- `teach` +- `undo`: The undo steps a person approves after a stop. + +**Give agents the program:** + +- `emit-mcp` +- `emit-skill` +- `serve-execute` + +**Proof:** + +- `attach-seal` +- `countersign` +- `local-seal` +- `seal` + +**Run on premises:** + +- `no-outbound` +- `service` + +**Share safely with OpenAdapt Cloud:** + +- `approve-sanitized` +- `connect` +- `connector` +- `login` +- `materialize-approved` +- `push` +- `report-break` +- `report-run` +- `review-sanitized` +- `sanitize` +- `validate-hosted` + +**Try it and measure it:** + +- `bench` +- `benchmark` +- `demo` +- `tutorial` + + ## Measured results @@ -1128,6 +1224,30 @@ halts with a named reason when the confirmation is missing before and after numbers are in [`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md). +Before a step stops on a look-alike identifier, the replayer takes extra looks +that change nothing +([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)): +the same read on two fresh screenshots, the 2x re-read, the known-ID list when +one is configured, and, where the surface allows it, a select-and-copy of the +identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle +the record on their own, because a look-alike twin can read the same on every +frame and at every size. Only two looks can: the known-ID list, and a copied +identifier that equals a run parameter in the identity band. A look that names +another record stops the run. Each look is recorded on the step's identity +check. + +The copy look has limits. It needs a surface where selecting the identifier +and copying it changes nothing, and a clipboard that reaches the machine Flow +runs on (RDP or Citrix clipboard redirection). It drags across only the one +text box that holds the identifier, and the run stops if the screen changes +anywhere else during the look. A canvas with no selectable text has no copy +look. A recording's own identifier is OCR text and can be a misread (the RDP +kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the +recorded record, the copy can stop a run but never confirm one. It confirms +only a record chosen by a run parameter. Governed and qualification runs never +take it. The check just before the click takes no input; it reuses the copy +only when the identifier's pixels are unchanged since the look. + Permissive `replay` may proceed on an unreadable reversible target and reports that condition. Governed `run` requires an affirmative live verdict for every identity-required admitted step; mismatch, unreadable, or abstain halts before @@ -1156,6 +1276,39 @@ irreversible, and qualification requires its record check. An icon-only click that only navigates after data entry is therefore over-classified until an operator reclassifies it. +On a pixel surface an icon-only click carries no text that names its control. +When the recorder can move the pointer without clicking +(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the +pointer on each click target first and keeps that frame. For a click with no +other identity evidence, the compiler stores what changed near the target, such +as the control's highlight or tooltip, as the step's hover evidence +([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)). +At replay, on the pixel rungs, the runtime rests the pointer on the resolved +target, waits the same 0.8 s, and compares. A different response, or a +response where the recording saw none, stops the run before the click. A +missing response, or one that can't be compared (another render with no +readable text), is recorded on the step and changes nothing. A tooltip that +opens later than the wait looks like a different response and stops the run. +Controls that don't answer the pointer, such as most text fields, get no +evidence this way. The browser, remote-display (Citrix and RDP client window) +and FreeRDP backends can hover. The Aardwolf RDP transport and the native +Windows, macOS and Linux backends can't move the pointer without a button +yet, so they take no hover look. + +A pixel step is checked on one frame and clicked a moment later. Consequential +identity-gated clicks were already bound to the exact checked frame. Every +other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`): +right before the click, the runtime takes a fresh frame and compares the +target region, the identity region and the hover response with the frame the +step was checked on. If one changed (a dialog opened, the list re-sorted), it +doesn't click. It re-resolves the target and re-runs the identity check on +the fresh frame, with no input, and clicks only when the next frame matches +that one. After two re-observations that still see a change, or when the +target or its identity can't be confirmed again, it refuses the click. Each +change is recorded on the step (`StepResult.check_to_click`). A change smaller +than 4 pixels, or under 24 levels per channel, doesn't count. Structured +targets are bound by element tokens instead and aren't rechecked. + A consequential click must be tied to a record before Flow delivers it. Inside a table row, the row is that record. An icon-only Save that floats beside a form (openIMIS's claim Save has no text, accessible name, or tooltip) has no diff --git a/packages/capture/README.md b/packages/capture/README.md index c38982e8b..a6a2bf108 100644 --- a/packages/capture/README.md +++ b/packages/capture/README.md @@ -165,17 +165,31 @@ encode-and-decode test and refuses to start if it fails. ## Microphone narration `capture record --audio` also records your voice and transcribes it on this -computer. It's off by default. It needs the `audio` extra, which adds the -microphone libraries and a local transcription backend (faster-whisper): +computer. It's off by default. It needs the `audio` extra (the microphone +libraries and faster-whisper) and the `privacy` extra, because the transcript +is scrubbed before it is stored: ```bash -pip install "openadapt-capture[audio]" +pip install "openadapt-capture[audio,privacy]" +python -m spacy download en_core_web_sm ``` -Capture refuses `--audio` before it opens the microphone if the extra or a -local backend isn't installed, and it deletes the audio after transcription unless you set +Capture refuses `--audio` before it opens the microphone if either is +missing, and it deletes the audio after transcription unless you set `RECORD_AUDIO_RETAIN_WAVEFORM`. +The scrubbed transcript is saved as `narration.json` in the capture folder: +one entry per sentence, timed on the same clock as your clicks and keys. Word +by word text is not kept. `capture narration ./my_recording` prints it, and +`capture transcribe ./my_recording` builds it from a retained `audio.flac`. + +OpenAdapt Flow uses these sentences when it learns a program from several +recordings. If you say why you take a branch ("if the insurance is Medicare, +skip the copay screen"), Flow proposes that condition. It becomes part of the +program only when every recording agrees with it, a test copy confirms it, or +a person accepts it. See Flow's +[NARRATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/NARRATION.md). + ## Privacy A recording can contain anything that was on screen or typed, including diff --git a/packages/capture/docs/COMPLIANCE.md b/packages/capture/docs/COMPLIANCE.md index b7f485757..bacc7dc8f 100644 --- a/packages/capture/docs/COMPLIANCE.md +++ b/packages/capture/docs/COMPLIANCE.md @@ -96,9 +96,19 @@ for egress. These boundaries are in code: - The HTML viewer doesn't embed audio by default, because that file is self-contained and easy to forward. - `--audio` prints a microphone notice before recording starts. - -Transcript text is unscrubbed free text. Treat it as at least as sensitive as -the rest of the capture. Nothing downstream reads it today. +- The transcript is scrubbed with openadapt-privacy before it is stored, in + the database and in `narration.json`. `--audio` and `capture transcribe` + refuse when openadapt-privacy is not installed, so there is no unscrubbed + fallback. +- Word-level text and timings are not stored. A name spoken one word at a + time can't be recognized by a text scrubber, so only whole sentences are + kept. + +Scrubbing is automatic and can miss things, so treat the scrubbed transcript +as sensitive, like the rest of the capture. OpenAdapt Flow reads +`narration.json` to propose branch conditions; it never sends the sentences +anywhere unless you opt in to `induce --narration-model`, which scrubs them +again first. ## Recorder control channel diff --git a/packages/capture/docs/DESIGN.md b/packages/capture/docs/DESIGN.md index 689f1ad3a..f647038d6 100644 --- a/packages/capture/docs/DESIGN.md +++ b/packages/capture/docs/DESIGN.md @@ -247,8 +247,14 @@ installed on-device transcription backend before it opens the microphone. It does not use a remote fallback. It discards the waveform after transcription unless the operator explicitly enables waveform retention. -Transcript text is unsanitized. A retained waveform is biometric data. Neither -is safe for automatic egress. +Capture also requires openadapt-privacy before it opens the microphone, and +scrubs the transcript before storing it. The scrubbed sentences go to +`narration.json` (`openadapt_capture.narration`), timed on the recording +clock so a consumer can line each one up with an action. Per-word text is not +stored. + +Scrubbing can miss data, so the transcript is still sensitive. A retained +waveform is biometric data. Neither is safe for automatic egress. ## Browser boundary diff --git a/packages/capture/openadapt_capture/capture.py b/packages/capture/openadapt_capture/capture.py index bcacdfaa8..30189f8f5 100644 --- a/packages/capture/openadapt_capture/capture.py +++ b/packages/capture/openadapt_capture/capture.py @@ -1322,6 +1322,16 @@ def desktop_capture(self) -> dict | None: return config.get("capture_desktop") return None + def narration(self) -> list | None: + """Scrubbed spoken narration as timed sentences, or None. + + Reads ``narration.json`` (see :mod:`openadapt_capture.narration`). + Times are on the recording clock, like action timestamps. + """ + from openadapt_capture.narration import read_narration + + return read_narration(self.capture_dir) + @property def audio_start_time(self) -> float | None: """Start timestamp of the audio recording, or None if unavailable.""" diff --git a/packages/capture/openadapt_capture/cli.py b/packages/capture/openadapt_capture/cli.py index cd8fb9af9..357b11af1 100644 --- a/packages/capture/openadapt_capture/cli.py +++ b/packages/capture/openadapt_capture/cli.py @@ -107,6 +107,11 @@ def record( except NoLocalTranscriptionBackend as exc: print(str(exc)) raise SystemExit(1) from exc + from openadapt_capture.narration import NO_SCRUBBER_MESSAGE, narration_scrubber + + if narration_scrubber() is None: + print(NO_SCRUBBER_MESSAGE) + raise SystemExit(1) retain = settings.RECORD_AUDIO_RETAIN_WAVEFORM print("MICROPHONE ON: this recording captures spoken narration.") @@ -117,8 +122,8 @@ def record( else " Waveform discarded after transcription; only text is retained." ) print( - " Anything said aloud is retained as text. Do not speak names, " - "dates of birth, or other identifying details." + " The text is scrubbed of patient data before it is stored. Still, " + "do not speak names, dates of birth, or other identifying details." ) print() @@ -334,11 +339,13 @@ def transcribe( model: str = "base", backend: str = "auto", ) -> None: - """Transcribe a capture's audio using an on-device Whisper model. + """Transcribe a capture's retained audio on this machine and store narration. Transcription runs entirely on this machine. There is no remote backend: a raw waveform cannot be sanitized before upload and discloses the - speaker's voice as well as their words. + speaker's voice as well as their words. The transcript is scrubbed with + openadapt-privacy and written to ``narration.json`` (timed sentences) and + ``transcript.txt``. Without the scrubber nothing is written. Args: capture_dir: Path to capture directory. @@ -352,11 +359,17 @@ def transcribe( NoLocalTranscriptionBackend, resolve_transcription_backend, ) + from openadapt_capture.narration import ( + NO_SCRUBBER_MESSAGE, + narration_scrubber, + scrub_utterances, + transcribe_file, + utterances_from_transcript, + write_narration, + ) capture_dir = Path(capture_dir) audio_path = capture_dir / "audio.flac" - transcript_path = capture_dir / "transcript.txt" - transcript_json_path = capture_dir / "transcript.json" if not audio_path.exists(): print(f"No audio file found at: {audio_path}") @@ -367,124 +380,54 @@ def transcribe( except (NoLocalTranscriptionBackend, ValueError) as exc: print(str(exc)) raise SystemExit(1) from exc + scrubber = narration_scrubber() + if scrubber is None: + print(NO_SCRUBBER_MESSAGE) + raise SystemExit(1) - print(f"Transcribing on-device with backend: {backend}") - - if backend == "faster-whisper": - _transcribe_faster_whisper(audio_path, transcript_path, transcript_json_path, model) - else: - _transcribe_local(audio_path, transcript_path, transcript_json_path, model) - - -def _transcribe_local( - audio_path: Path, - transcript_path: Path, - transcript_json_path: Path, - model: str, -) -> None: - """Transcribe using local openai-whisper model.""" - - print(f"Transcribing audio with openai-whisper ({model} model)...") - print("This may take a moment...") - + offset = 0.0 try: - import whisper - except ImportError: - print("Whisper not installed. Install with: pip install openai-whisper") - print("Or use faster-whisper: pip install faster-whisper") - return - - # Load model and transcribe with word timestamps - whisper_model = whisper.load_model(model) - result = whisper_model.transcribe(str(audio_path), fp16=False, word_timestamps=True) - - transcript = result.get("text", "").strip() - - # Extract segments with timestamps - segments = [] - for segment in result.get("segments", []): - segments.append({ - "start": segment["start"], - "end": segment["end"], - "text": segment["text"].strip(), - }) - - # Save transcripts - _save_transcript(transcript, segments, transcript_path, transcript_json_path) - - -def _transcribe_faster_whisper( - audio_path: Path, - transcript_path: Path, - transcript_json_path: Path, - model: str, -) -> None: - """Transcribe using faster-whisper (4x faster than openai-whisper).""" + from openadapt_capture.capture import CaptureSession - print(f"Transcribing audio with faster-whisper ({model} model)...") - print("This is 4x faster than openai-whisper...") + offset = CaptureSession.load(capture_dir).audio_start_time or 0.0 + except Exception: # noqa: BLE001 - a bare audio folder has no recording clock + offset = 0.0 - try: - from faster_whisper import WhisperModel - except ImportError: - print("faster-whisper not installed. Install with: pip install faster-whisper") - print("Or use openai-whisper: pip install openai-whisper") - return - - # Load model with CPU and int8 for efficiency - whisper_model = WhisperModel(model, device="cpu", compute_type="int8") - - # Transcribe with word timestamps - segments_iter, info = whisper_model.transcribe( - str(audio_path), - word_timestamps=True, + print(f"Transcribing on-device with backend: {backend}") + result = transcribe_file(audio_path, backend=backend, model=model) + utterances = scrub_utterances( + utterances_from_transcript(result, offset=offset), scrubber ) + path = write_narration( + capture_dir, + utterances, + scrubber=scrubber, + transcription_backend=backend, + audio_start=offset, + ) + transcript_path = capture_dir / "transcript.txt" + transcript_path.write_text( + " ".join(u.text for u in utterances) + "\n", encoding="utf-8" + ) + print(f"Saved scrubbed narration to: {path}") + print(f"Saved scrubbed transcript to: {transcript_path}") + for u in utterances: + rel = u.start - offset + print(f"[{int(rel // 60)}:{rel % 60:05.2f}] {u.text}") - # Collect segments - segments = [] - full_text_parts = [] - for segment in segments_iter: - segments.append({ - "start": segment.start, - "end": segment.end, - "text": segment.text.strip(), - }) - full_text_parts.append(segment.text) - - transcript = "".join(full_text_parts).strip() - - # Save transcripts - _save_transcript(transcript, segments, transcript_path, transcript_json_path) - - -def _save_transcript( - transcript: str, - segments: list[dict], - transcript_path: Path, - transcript_json_path: Path, -) -> None: - """Save transcript to files and print output.""" - import json - - # Save plain text transcript - transcript_path.write_text(transcript, encoding="utf-8") - # Save JSON with timestamps - transcript_json_path.write_text( - json.dumps({"text": transcript, "segments": segments}, indent=2), - encoding="utf-8" - ) +def narration(capture_dir: str) -> None: + """Print a capture's scrubbed narration, one timed sentence per line.""" + from openadapt_capture.narration import read_narration - print(f"Saved transcript to: {transcript_path}") - print(f"Saved timestamps to: {transcript_json_path}") - print() - print("Transcript:") - print("-" * 40) - for seg in segments: - start = seg["start"] - mins = int(start // 60) - secs = start % 60 - print(f"[{mins}:{secs:05.2f}] {seg['text']}") + utterances = read_narration(capture_dir) + if utterances is None: + print(f"No narration.json in {capture_dir}") + return + first = utterances[0].start if utterances else 0.0 + for u in utterances: + rel = u.start - first + print(f"[{int(rel // 60)}:{rel % 60:05.2f}] {u.text}") def install_ffmpeg( @@ -624,6 +567,7 @@ def main() -> None: "visualize": visualize, "info": info, "transcribe": transcribe, + "narration": narration, "share": share, "install-ffmpeg": install_ffmpeg, "uninstall-ffmpeg": uninstall_ffmpeg, diff --git a/packages/capture/openadapt_capture/narration.py b/packages/capture/openadapt_capture/narration.py new file mode 100644 index 000000000..25586ccc7 --- /dev/null +++ b/packages/capture/openadapt_capture/narration.py @@ -0,0 +1,294 @@ +"""Spoken narration beside a recording: utterances, scrubbing and the sidecar. + +People explain their choices out loud while they demonstrate ("the insurance +is Medicare, so I skip the copay screen"). This module turns an on-device +transcript into timed **utterances**, scrubs patient data out of them with +``openadapt-privacy``, and writes them next to the recording as +``narration.json``. ``openadapt-flow`` reads that file, lines each utterance up +with the recorded steps, and uses it to *propose* (never certify) a branch rule. + +Rules this module enforces: + +1. **Scrub before storing.** :func:`scrub_utterances` runs every utterance + through the scrubber before anything is written. With no scrubber (the + ``privacy`` extra is missing) it raises :class:`NarrationNotScrubbed`, and + the recorder stores no transcript at all. There is no plaintext fallback. +2. **Sentences, not words.** Word-level timings are used only to cut a segment + into sentences, then dropped. A name spoken one word at a time cannot be + recognised by a text scrubber, so per-word text is never stored. +3. **Local only.** :func:`transcribe_file` uses the same on-device backends as + :mod:`openadapt_capture.audio`. Nothing is uploaded. + +Timestamps in ``narration.json`` are on the recording's clock (the same clock +as action events), so an utterance can be matched to the step it explains on +any backend: web, native, RDP or Citrix. +""" + +from __future__ import annotations + +import json +import os +import re +import tempfile +from dataclasses import asdict, dataclass +from importlib import metadata +from pathlib import Path +from typing import Any, Optional, Protocol, Sequence, runtime_checkable + +NARRATION_FILE = "narration.json" +NARRATION_SCHEMA = "openadapt.capture.narration/v1" + +_SENTENCE_END = re.compile(r"[.?!]['\")\]]*$") + +NO_SCRUBBER_MESSAGE = ( + "Narration is not stored without scrubbing. Spoken words can name a " + "patient, and openadapt-privacy is not installed to remove that. Install " + "it with:\n" + ' pip install "openadapt-capture[audio,privacy]"\n' + " python -m spacy download en_core_web_sm" +) + + +class NarrationNotScrubbed(RuntimeError): + """The transcript could not be scrubbed, so it was not stored.""" + + def __init__(self, message: str = NO_SCRUBBER_MESSAGE) -> None: + super().__init__(message) + + +@runtime_checkable +class TextScrubber(Protocol): + """The part of an ``openadapt-privacy`` provider narration needs.""" + + def scrub_text(self, text: str, is_separated: bool = False) -> str: ... + + +@dataclass(frozen=True) +class Utterance: + """One spoken sentence, timed on the recording's clock (seconds).""" + + start: float + end: float + text: str + + +def narration_scrubber() -> Optional[TextScrubber]: + """The scrubber narration uses: the same provider ``share send`` uses.""" + from openadapt_capture.share_scrub import get_scrubber + + return get_scrubber() + + +def _scrubber_evidence(scrubber: TextScrubber) -> dict[str, Any]: + try: + version = metadata.version("openadapt-privacy") + except metadata.PackageNotFoundError: + version = None + return { + "class": f"{type(scrubber).__module__}.{type(scrubber).__qualname__}", + "openadapt_privacy_version": version, + } + + +def utterances_from_transcript( + result: dict[str, Any], *, offset: float = 0.0 +) -> list[Utterance]: + """Split a Whisper-style result into sentence utterances. + + ``result`` is what :meth:`openadapt_capture.audio.AudioRecorder.transcribe` + returns: ``{"segments": [{"start", "end", "text", "words"?}]}`` with times + relative to the start of the audio. ``offset`` (the audio's start time on + the recording clock) is added to every time. Every segment is used, not + only the first. When a segment has word timings it is cut at words that + end a sentence; otherwise the whole segment is one utterance. + """ + out: list[Utterance] = [] + for segment in result.get("segments") or []: + words = segment.get("words") or [] + if words: + current: list[dict[str, Any]] = [] + for word in words: + current.append(word) + if _SENTENCE_END.search(str(word.get("word", "")).strip()): + out.append(_join(current, offset)) + current = [] + if current: + out.append(_join(current, offset)) + continue + text = str(segment.get("text") or "").strip() + if text: + out.append( + Utterance( + start=round(float(segment.get("start", 0.0)) + offset, 3), + end=round(float(segment.get("end", 0.0)) + offset, 3), + text=text, + ) + ) + if not out and str(result.get("text") or "").strip(): + # No segments (some engines, or a stub): one utterance at the start. + out.append(Utterance(start=offset, end=offset, text=str(result["text"]).strip())) + return [u for u in out if u.text] + + +def _join(words: Sequence[dict[str, Any]], offset: float) -> Utterance: + text = "".join(str(w.get("word", "")) for w in words).strip() + text = re.sub(r"\s+", " ", text) + return Utterance( + start=round(float(words[0].get("start", 0.0)) + offset, 3), + end=round(float(words[-1].get("end", 0.0)) + offset, 3), + text=text, + ) + + +def scrub_utterances( + utterances: Sequence[Utterance], scrubber: Optional[TextScrubber] +) -> list[Utterance]: + """Return the utterances with patient data scrubbed out. + + Raises :class:`NarrationNotScrubbed` when there is no scrubber or it + fails. The caller must then store nothing. + """ + if scrubber is None: + raise NarrationNotScrubbed() + out: list[Utterance] = [] + for u in utterances: + try: + text = scrubber.scrub_text(u.text) + except Exception as exc: # noqa: BLE001 - any failure means not scrubbed + raise NarrationNotScrubbed( + f"The scrubber failed on the transcript, so it was not stored: {exc}" + ) from exc + out.append(Utterance(start=u.start, end=u.end, text=str(text or "").strip())) + return out + + +def write_narration( + directory: str | Path, + utterances: Sequence[Utterance], + *, + scrubber: TextScrubber, + transcription_backend: Optional[str] = None, + audio_start: Optional[float] = None, +) -> Path: + """Write already-scrubbed utterances to ``/narration.json``. + + The file says which scrubber ran. It is written atomically. + """ + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + payload = { + "schema": NARRATION_SCHEMA, + "clock": "recording", + "audio_start": audio_start, + "transcription": {"backend": transcription_backend, "on_device": True}, + "scrubbed": True, + "scrubbed_by": _scrubber_evidence(scrubber), + "utterances": [asdict(u) for u in utterances], + } + path = directory / NARRATION_FILE + fd, tmp = tempfile.mkstemp(dir=directory, prefix=".narration-", suffix=".json") + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2) + handle.write("\n") + os.replace(tmp, path) + except BaseException: + Path(tmp).unlink(missing_ok=True) + raise + return path + + +def read_narration(directory: str | Path) -> Optional[list[Utterance]]: + """Read ``narration.json`` from a capture directory, or ``None``. + + Refuses (``ValueError``) a file that does not say it was scrubbed. + """ + path = Path(directory) / NARRATION_FILE + if not path.is_file(): + return None + data = json.loads(path.read_text(encoding="utf-8")) + if data.get("schema") != NARRATION_SCHEMA: + raise ValueError(f"{path} is not a narration file ({data.get('schema')!r})") + if data.get("scrubbed") is not True: + raise ValueError(f"{path} does not say it was scrubbed; refusing to read it") + return [ + Utterance(start=float(u["start"]), end=float(u["end"]), text=str(u["text"])) + for u in data.get("utterances") or [] + ] + + +SAMPLE_RATE = 16000 + + +def load_waveform(path: str | Path) -> Any: + """Read a 16 kHz audio file as a mono float32 array (what Whisper takes). + + WAV is read with the standard library; other formats (FLAC) need + ``soundfile`` from the ``audio`` extra. Other sample rates are refused + rather than resampled silently. + """ + import numpy as np + + path = Path(path) + if path.suffix.lower() == ".wav": + import wave + + with wave.open(str(path), "rb") as handle: + rate = handle.getframerate() + channels = handle.getnchannels() + width = handle.getsampwidth() + raw = handle.readframes(handle.getnframes()) + if width != 2: + raise ValueError(f"{path}: only 16-bit PCM WAV is supported") + data = np.frombuffer(raw, dtype=" 1: + data = data.reshape(-1, channels).mean(axis=1) + else: + import soundfile + + data, rate = soundfile.read(str(path), dtype="float32", always_2d=True) + data = data.mean(axis=1) + if rate != SAMPLE_RATE: + raise ValueError(f"{path}: sample rate {rate} Hz; narration needs 16000 Hz") + return np.ascontiguousarray(data, dtype=np.float32) + + +def transcribe_file( + path: str | Path, *, backend: str = "auto", model: str = "base" +) -> dict[str, Any]: + """Transcribe an audio file on this machine (no network recognizer). + + Returns a Whisper-style result with word timings. Raises + :class:`~openadapt_capture.audio.NoLocalTranscriptionBackend` when no local + backend is installed. + """ + from openadapt_capture.audio import resolve_transcription_backend + + chosen = resolve_transcription_backend(backend) + waveform = load_waveform(path) + if chosen == "faster-whisper": + from faster_whisper import WhisperModel + + engine = WhisperModel(model, device="cpu", compute_type="int8") + segments, info = engine.transcribe(waveform, word_timestamps=True) + out = [] + for seg in segments: + item: dict[str, Any] = {"start": seg.start, "end": seg.end, "text": seg.text} + if seg.words: + item["words"] = [ + {"word": w.word, "start": w.start, "end": w.end} for w in seg.words + ] + out.append(item) + return { + "text": "".join(s["text"] for s in out), + "segments": out, + "language": info.language, + "backend": chosen, + } + import whisper + + result = whisper.load_model(model).transcribe( + waveform, word_timestamps=True, fp16=False + ) + result["backend"] = chosen + return result diff --git a/packages/capture/openadapt_capture/recorder.py b/packages/capture/openadapt_capture/recorder.py index 0e3e44379..c257c369c 100644 --- a/packages/capture/openadapt_capture/recorder.py +++ b/packages/capture/openadapt_capture/recorder.py @@ -2698,6 +2698,10 @@ def record_audio( text is retained by default. - The transcript is never logged. Spoken narration may contain identifying details and must not be copied into logs or terminal scrollback. + - The transcript is scrubbed with openadapt-privacy before it is stored. + Without a scrubber this refuses before the microphone opens. The scrubbed + sentences are written to ``narration.json`` beside the database, timed on + the recording clock; per-word text and timings are not stored. Args: recording: The recording object. @@ -2716,6 +2720,13 @@ def record_audio( # operator for nothing and failing the capture at the end. require_audio_dependencies() backend = require_local_transcription_backend() + # The transcript is scrubbed before it is stored. Without a scrubber there + # is nothing safe to store, so refuse before the microphone opens too. + from openadapt_capture.narration import NarrationNotScrubbed, narration_scrubber + + scrubber = narration_scrubber() + if scrubber is None: + raise NarrationNotScrubbed() utils.set_start_time(recording.timestamp) @@ -2802,13 +2813,34 @@ def audio_callback( "Transcription complete ({} characters).".format(len(result_info.get("text") or "")) ) - # empty word_list if the user didn't say anything - word_list = [] - # segments could be empty - if len(result_info["segments"]) > 0: - # there won't be a 'words' list if the user didn't say anything - if "words" in result_info["segments"][0]: - word_list = result_info["segments"][0]["words"] + # Cut the transcript into timed sentences and scrub each one before + # anything is stored. Word-level timings are dropped: a name spoken one + # word at a time cannot be recognised by a text scrubber. + from pathlib import Path + + from openadapt_capture.narration import ( + scrub_utterances, + utterances_from_transcript, + write_narration, + ) + + try: + utterances = scrub_utterances( + utterances_from_transcript(result_info, offset=start_timestamp), scrubber + ) + except NarrationNotScrubbed as exc: + logger.error("Narration not stored: {}", type(exc).__name__) + utterances = None + transcript = " ".join(u.text for u in utterances) if utterances else "" + word_list: list = [] + if utterances and db_path not in (None, ":memory:"): + write_narration( + Path(db_path).parent, + utterances, + scrubber=scrubber, + transcription_backend=backend, + audio_start=start_timestamp, + ) if config.RECORD_AUDIO_RETAIN_WAVEFORM: # Explicitly opted in. The retained waveform is biometric identifying @@ -2835,7 +2867,7 @@ def audio_callback( crud.insert_audio_info( session, compressed_audio_bytes, - result_info["text"], + transcript, recording, start_timestamp, sample_rate, diff --git a/packages/capture/tests/fixtures/narration/medicare_copay.wav b/packages/capture/tests/fixtures/narration/medicare_copay.wav new file mode 100644 index 000000000..cff2c1e51 Binary files /dev/null and b/packages/capture/tests/fixtures/narration/medicare_copay.wav differ diff --git a/packages/capture/tests/test_audio_privacy.py b/packages/capture/tests/test_audio_privacy.py index dea53459c..bf670c1f9 100644 --- a/packages/capture/tests/test_audio_privacy.py +++ b/packages/capture/tests/test_audio_privacy.py @@ -37,6 +37,22 @@ PACKAGE_ROOT = Path(audio_mod.__file__).resolve().parent +class _UpperScrubber: + """A stand-in scrubber: replaces the word 'patient' so tests can see it ran.""" + + def scrub_text(self, text: str, is_separated: bool = False) -> str: + return text.replace("patient", "") + + +@pytest.fixture(autouse=True) +def _scrubber(): + from openadapt_capture import share_scrub + + share_scrub.set_scrubber(_UpperScrubber()) + yield + share_scrub.reset_scrubber() + + # --------------------------------------------------------------------------- # 1. On-device only: no network recognizer, and no silent fallback to one. # --------------------------------------------------------------------------- @@ -191,8 +207,10 @@ def test_record_audio_preflight_precedes_stream_construction_in_source() -> None source = inspect.getsource(recorder_mod.record_audio) preflight = source.index("require_local_transcription_backend()") + scrubber = source.index("raise NarrationNotScrubbed()") stream = source.index("InputStream(") assert preflight < stream + assert scrubber < stream # --------------------------------------------------------------------------- @@ -306,7 +324,9 @@ def test_waveform_is_discarded_by_default(monkeypatch) -> None: captured = _run_record_audio(monkeypatch, retain=False) assert captured["audio_data"] == b"", "waveform retained without opt-in" - assert captured["text"] == "patient name spoken aloud" + # The stored transcript is the scrubbed one; per-word text is withheld. + assert captured["text"] == " name spoken aloud" + assert captured["words"] == [] def test_waveform_is_retained_only_on_explicit_opt_in(monkeypatch) -> None: diff --git a/packages/capture/tests/test_narration.py b/packages/capture/tests/test_narration.py new file mode 100644 index 000000000..c25d20f87 --- /dev/null +++ b/packages/capture/tests/test_narration.py @@ -0,0 +1,237 @@ +"""Narration beside a recording: timed sentences, scrubbed before storing. + +The audio fixture ``fixtures/narration/medicare_copay.wav`` is synthetic +speech made on macOS with ``say`` ("Patient John Smith. If the insurance is +Medicare, skip the copay screen."). The name in it is invented. +""" + +from __future__ import annotations + +import importlib.util +import json +import multiprocessing +import sys +import threading +import time +import types +from pathlib import Path + +import pytest + +from openadapt_capture import audio as audio_mod +from openadapt_capture import share_scrub +from openadapt_capture.narration import ( + NARRATION_FILE, + NarrationNotScrubbed, + Utterance, + load_waveform, + read_narration, + scrub_utterances, + utterances_from_transcript, + write_narration, +) + +FIXTURE = Path(__file__).parent / "fixtures" / "narration" / "medicare_copay.wav" + + +class NameScrubber: + """Stand-in for openadapt-privacy: removes the one invented name.""" + + def scrub_text(self, text: str, is_separated: bool = False) -> str: + return text.replace("John Smith", "") + + +@pytest.fixture() +def scrubber(): + share_scrub.set_scrubber(NameScrubber()) + yield share_scrub.get_scrubber() + share_scrub.reset_scrubber() + + +def test_every_segment_is_cut_into_timed_sentences(): + result = { + "segments": [ + { + "start": 0.0, + "end": 4.0, + "text": " Open the chart. If it's Medicare, skip copay.", + "words": [ + {"word": " Open", "start": 0.0, "end": 0.4}, + {"word": " the", "start": 0.4, "end": 0.6}, + {"word": " chart.", "start": 0.6, "end": 1.0}, + {"word": " If", "start": 1.5, "end": 1.7}, + {"word": " it's", "start": 1.7, "end": 2.0}, + {"word": " Medicare,", "start": 2.0, "end": 2.6}, + {"word": " skip", "start": 2.8, "end": 3.1}, + {"word": " copay.", "start": 3.1, "end": 3.6}, + ], + }, + {"start": 6.0, "end": 7.0, "text": " Now save."}, + ] + } + utts = utterances_from_transcript(result, offset=100.0) + assert utts == [ + Utterance(100.0, 101.0, "Open the chart."), + Utterance(101.5, 103.6, "If it's Medicare, skip copay."), + Utterance(106.0, 107.0, "Now save."), + ] + + +def test_scrubbing_is_required_and_removes_the_name(): + utts = [Utterance(0.0, 1.0, "Patient John Smith, insurance Medicare.")] + with pytest.raises(NarrationNotScrubbed): + scrub_utterances(utts, None) + (clean,) = scrub_utterances(utts, NameScrubber()) + assert "John Smith" not in clean.text + assert "Medicare" in clean.text + + +def test_sidecar_records_the_scrubber_and_refuses_unscrubbed_files(tmp_path): + utts = scrub_utterances([Utterance(1.0, 2.0, "John Smith is here.")], NameScrubber()) + path = write_narration(tmp_path, utts, scrubber=NameScrubber(), audio_start=1.0) + data = json.loads(path.read_text()) + assert data["scrubbed"] is True + assert data["scrubbed_by"]["class"].endswith("NameScrubber") + assert "John Smith" not in path.read_text() + assert read_narration(tmp_path) == utts + + data["scrubbed"] = False + path.write_text(json.dumps(data)) + with pytest.raises(ValueError, match="scrubbed"): + read_narration(tmp_path) + + +class _Stream: + samplerate = 16000 + + def __init__(self, callback=None, samplerate=16000, channels=1): + self._cb = callback + self._stop = threading.Event() + + def start(self): + import numpy as np + + def loop(): + while not self._stop.is_set(): + self._cb(np.zeros((160, 1), dtype=np.float32), 160, None, None) + time.sleep(0.005) + + threading.Thread(target=loop, daemon=True).start() + + def stop(self): + self._stop.set() + + def close(self): + pass + + +def _record(monkeypatch, tmp_path, transcript): + from openadapt_capture import recorder as recorder_mod + + fake_sd = types.ModuleType("sounddevice") + fake_sd.InputStream = _Stream + fake_sd.CallbackFlags = object + monkeypatch.setitem(sys.modules, "sounddevice", fake_sd) + monkeypatch.setattr(audio_mod, "_get_best_transcription_backend", lambda: "faster-whisper") + monkeypatch.setattr(recorder_mod, "_transcribe_on_device", lambda a, b: transcript) + stored: dict = {} + monkeypatch.setattr( + recorder_mod.crud, + "insert_audio_info", + lambda s, a, text, r, ts, rate, words: stored.update(text=text, words=words, ts=ts), + ) + monkeypatch.setattr(recorder_mod, "get_session_for_path", lambda p: None) + terminate = multiprocessing.Event() + threading.Timer(0.1, terminate.set).start() + recorder_mod.record_audio( + recording=types.SimpleNamespace(timestamp=time.time()), + db_path=str(tmp_path / "recording.db"), + terminate_processing=terminate, + started_event=multiprocessing.Event(), + ) + return stored + + +def test_recorded_transcript_is_scrubbed_before_it_is_stored(monkeypatch, tmp_path, scrubber): + transcript = { + "text": "Patient John Smith. If the insurance is Medicare, skip the copay screen.", + "segments": [ + {"start": 0.2, "end": 1.6, "text": " Patient John Smith."}, + {"start": 2.0, "end": 4.4, "text": " If the insurance is Medicare, skip the copay screen."}, + ], + } + stored = _record(monkeypatch, tmp_path, transcript) + assert "John Smith" not in stored["text"] + assert stored["words"] == [] + sidecar = (tmp_path / NARRATION_FILE).read_text() + assert "John Smith" not in sidecar + utts = read_narration(tmp_path) + assert [u.text for u in utts] == [ + "Patient .", + "If the insurance is Medicare, skip the copay screen.", + ] + # Times are on the recording clock: audio start plus the segment offset. + assert utts[1].start == pytest.approx(stored["ts"] + 2.0, abs=1e-3) + + +def test_no_scrubber_means_the_microphone_never_opens(monkeypatch, tmp_path): + from openadapt_capture import recorder as recorder_mod + + share_scrub.set_scrubber(None) + opened: list[str] = [] + + class _Never: + def __init__(self, *a, **k): + opened.append("mic") + + fake_sd = types.ModuleType("sounddevice") + fake_sd.InputStream = _Never + fake_sd.CallbackFlags = object + monkeypatch.setitem(sys.modules, "sounddevice", fake_sd) + monkeypatch.setattr(audio_mod, "_get_best_transcription_backend", lambda: "faster-whisper") + try: + with pytest.raises(NarrationNotScrubbed): + recorder_mod.record_audio( + recording=object(), + db_path=str(tmp_path / "recording.db"), + terminate_processing=multiprocessing.Event(), + started_event=multiprocessing.Event(), + ) + finally: + share_scrub.reset_scrubber() + assert opened == [] + assert not (tmp_path / NARRATION_FILE).exists() + + +def test_fixture_is_16khz_synthetic_speech(): + wave = load_waveform(FIXTURE) + assert wave.dtype.name == "float32" + assert 2.0 < len(wave) / 16000 < 8.0 + assert float(abs(wave).max()) > 0.05 + + +@pytest.mark.skipif( + importlib.util.find_spec("faster_whisper") is None, + reason="faster-whisper (audio extra) not installed", +) +def test_local_whisper_transcribes_the_fixture(scrubber): + """End to end on this machine: synthetic speech -> local Whisper -> scrub. + + Needs the ``tiny.en`` model in the local Hugging Face cache; skipped when + it is not there, so the test never downloads anything. + """ + try: + from huggingface_hub import try_to_load_from_cache + + cached = try_to_load_from_cache("Systran/faster-whisper-tiny.en", "model.bin") + except Exception: # noqa: BLE001 + cached = None + if not isinstance(cached, str): + pytest.skip("tiny.en Whisper model is not cached locally") + from openadapt_capture.narration import transcribe_file + + result = transcribe_file(FIXTURE, backend="faster-whisper", model=str(Path(cached).parent)) + utts = scrub_utterances(utterances_from_transcript(result), scrubber) + text = " ".join(u.text for u in utts).lower() + assert "medicare" in text and "copay" in text + assert "john smith" not in text diff --git a/packages/flow/docs/CAPABILITIES.md b/packages/flow/docs/CAPABILITIES.md index 3bfa622f2..07459a1ec 100644 --- a/packages/flow/docs/CAPABILITIES.md +++ b/packages/flow/docs/CAPABILITIES.md @@ -72,6 +72,9 @@ at replaying one. These capabilities layer onto the same $0, model-free runtime. underdetermined. The healthy replay path still makes no model call. [`LEARN_FROM_MANY_RUNS.md`](LEARN_FROM_MANY_RUNS.md) covers `induce`, `teach-branch` and shadow mining together, with each one's status. + Spoken narration recorded with the demonstration proposes branch + conditions; a proposal becomes a guard only when every recording agrees, a + probe confirms it, or a person accepts it. See [`NARRATION.md`](NARRATION.md). With `--probe-on test-copy.yaml`, a refused optional step is instead replayed both ways on a declared test copy and scored by reading the record back; it is certified only when exactly one reading works for every diff --git a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md index a4eec454c..d828fd91b 100644 --- a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md +++ b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md @@ -45,6 +45,12 @@ step dropped and with it kept, scores each run by reading the record back, and certifies the step only when exactly one reading works for every recording. See [PROBE_INDUCTION.md](PROBE_INDUCTION.md). +If the person said why while recording ("if the insurance is Medicare, skip +the copay screen"), the sentence proposes the condition. It becomes a guard +only when every recording agrees with it, a probe confirms it, or a person +accepts it; a recording that contradicts it keeps the point refused. See +[NARRATION.md](NARRATION.md). + `admit rec-1 rec-2` runs the same induction as its first step, then proposes the checks and stops once for a decision (see [SHADOW_CANDIDATES.md](SHADOW_CANDIDATES.md#how-a-candidate-is-admitted)). @@ -62,7 +68,9 @@ which may be a name or a reference number. `teach-branch` needs at least two halts at the same step, fixed with the same corrective steps. It keys the branch on screen text that every example showed and no earlier screen did. If a normal run also shows that text, it narrows the guard with a run -parameter that separates the two cases, or refuses when none does. +parameter that separates the two cases, or refuses when none does. A +narrated sentence that names the parameter settles a tie between several, if +every run agrees with it ([NARRATION.md](NARRATION.md#teach-branch)). The result goes through the same learning loop as any repair (induction, regression gate, held-out check) and is written as a repair candidate. It is diff --git a/packages/flow/docs/NARRATION.md b/packages/flow/docs/NARRATION.md new file mode 100644 index 000000000..f5081472d --- /dev/null +++ b/packages/flow/docs/NARRATION.md @@ -0,0 +1,178 @@ +# Narration: why a branch was taken + +People often say why they do something while they demonstrate it: "if the +insurance is Medicare, skip the copay screen." When `induce` or `teach-branch` +finds a branch, the recordings show *that* the person did something different, +but not *why*. OpenAdapt uses the spoken sentence to propose the condition. + +A narrated condition is only a proposal. It becomes a guard in the program +only when one of these holds: + +- every recording agrees with it; +- `induce --probe-on` confirms it on a declared test copy + ([PROBE_INDUCTION.md](PROBE_INDUCTION.md)); +- a person accepts it (`induce --accept-narrated`). + +If a recording contradicts it, the point stays refused, as it would without +narration. At run time the guard is an ordinary predicate (a string compare on +a run parameter, or an OCR check for text on screen). Runs make no model calls. + +## Getting narration into a recording + +Narration is stored beside a recording as `narration.json`: scrubbed +sentences, each timed from the start of the recording and lined up with a +step. The file is the same for web, native, RDP and Citrix recordings, because +it only uses the `t` that every line in `events.jsonl` carries. + +There are two ways to get it. + +**Record with openadapt-capture.** `capture record --audio` transcribes on +this computer, scrubs the text with openadapt-privacy, and writes +`narration.json` in the capture folder. Converting the capture to a Flow +recording carries the sentences over and lines them up with the steps. + +```bash +pip install "openadapt-capture[audio,privacy]" +python -m spacy download en_core_web_sm +capture record ./visit-1 --audio +``` + +**Attach audio or a transcript to any recording.** Use this for recordings +made by Flow's own recorder (web), or for RDP and Citrix sessions recorded +alongside a separate microphone track. + +```bash +openadapt-flow narration attach ./recordings/visit-1 --audio visit-1.wav +openadapt-flow narration attach ./recordings/visit-1 --transcript visit-1.json --offset 2.5 +openadapt-flow narration show ./recordings/visit-1 +``` + +`--offset` is how many seconds after the recording started the audio starts. +Audio must be 16 kHz. Transcription uses openadapt-capture's on-device Whisper +backends and never uploads audio. + +A sentence is lined up with the next step the person takes after they start +speaking (people say what they're about to do), or with the last step if they +speak after it. + +## Privacy + +- **Scrubbed or not written.** Capture refuses `--audio` before the microphone + opens when openadapt-privacy is not installed. `narration attach` refuses + before transcribing. Neither has a plaintext fallback. +- **Sentences only.** Word timings are used to cut sentences and then dropped, + because a name spoken one word at a time can't be recognized by a text + scrubber. +- **Not shared off the machine by `visualize`.** The `remote-safe`, + `public-synthetic` and `sanitized-derivative` profiles drop the sentence and + keep only what certified the guard. +- **The compile-time model is opt-in.** `induce --narration-model` sends a + sentence the built-in parser can't read to Anthropic on your own key. The + text is scrubbed again on the way out, or the call is refused. The model's + answer can only name a recorded field and a value some recording had, and it + is checked against the recordings like any other proposal. + +## How a sentence becomes a guard + +1. **Find the sentence.** For each branch point, induction looks at sentences + lined up with the steps around it in every recording, then at sentences + elsewhere whose action names the step ("skip the copay screen" names a step + labeled "copay"). +2. **Read it.** The parser handles the usual shapes: "If X is Y, skip Z", + "Skip Z when X is Y", "Since it's Y, I don't fill in Z", "X is Y, so I skip + Z", "Unless X is Y, leave Z blank". "Skip", "don't", "unless" and "is not" + flip the condition. +3. **Map it.** If X names a recorded field that became a run parameter (for + example `insurance`) and Y is a value some recording typed, the guard is + `param_equals(insurance, "Medicare")`. Otherwise it is + `text_present("Medicare")`: the value must be on screen. +4. **Check it.** In each recording, induction evaluates the guard and compares + it with what the person did. A parameter guard uses the value that + recording typed, compared exactly as the runtime compares it. A screen guard uses OCR of the recording's frame at that + point. + +| Result | What happens | +| --- | --- | +| Every recording agrees | Certified. The step (or path) runs only when the guard holds. | +| A recording contradicts it | Refused, as without narration. The report names the recording. | +| It can't be checked in some recording (no frame, no OCR) | Proposed. A probe or a person can certify it. | + +Two shapes of branch are supported: + +- **An optional step** that some recordings have and others don't. The step + gets the narrated guard and is skipped when it does not hold. +- **A divergence** where recordings take one of two different paths at the + same point. The program gets a branch: the path the sentence names when the + guard holds, the other path otherwise. A divergence with another step inside + it stays refused. + +## Example + +Three recordings of checking in a visit. The Medicare one has no copay step, +and the person said "If the insurance is Medicare, skip the copay screen." + +```text +$ openadapt-flow induce rec-medicare rec-aetna rec-cigna --out ./checkin +Induction over 3 trace(s): 3 inferred, 0 proposed (flagged), 0 underdetermined. + [inferred] param 'insurance' -- 'insurance' varies across traces ['Medicare', 'Aetna', 'Cigna'] + [inferred] guard at opt_2: run 'type copay' only when insurance is not 'Medicare' -- suggested by narration, certified: every recording agrees + [narrated] opt_2: 'If the insurance is Medicare, skip the copay screen.' -> run 'type copay' only when insurance is not 'Medicare' -- CERTIFIED (every recording agrees) +Certification: CERTIFIED +``` + +Add a fourth recording where the person entered a copay for a Medicare +patient, and the guard is refused: + +```text + [narrated] opt_2: ... -- REFUSED (recording 3 contradicts it) +Certification: NOT CERTIFIED -- resolve the point(s) above. +``` + +When the guard could not be checked, a person can accept it. The report +prints each proposal as `LOCATION@GUARD_ID`; accepting by that id pins the +exact guard you read, so a different guard proposed later at the same place is +not accepted. Their name is stored with the guard: + +```bash +openadapt-flow induce rec-a rec-b --out ./prog --accept-narrated opt_2@3fa9c1d2e4b5 --accepted-by "R. Reviewer" +``` + +A probe result is pinned the same way. + +## Explaining the program back + +A certified guard keeps its sentence in the bundle (`Workflow.guard_sources`, +keyed by program state). The runtime never reads it. + +- `openadapt-flow visualize BUNDLE` shows a "narrated rule" badge on the step + or branch, and in the step's details: *because "If the insurance is + Medicare, skip the copay screen." (said while recording; certified: every + recording agrees)*. The Mermaid output lists the sentences as comments. +- `openadapt-flow repair show CANDIDATE` lists the narrated guards of a + taught candidate with their sentences. +- The `induce` report (`--report`) lists every narrated proposal, what it was + checked against, and how it was decided. + +## teach-branch + +`teach-branch` reads `narration.json` beside each fix recording and halted +run. When a sentence names an enum or boolean parameter ("if the payer type +is medicaid, dismiss the notice"), teach-branch narrows the branch to that +parameter, but only if every example has that value and every normal run that +shows the same notice has a different one. That settles the case where +several parameters separate the examples and teach-branch would otherwise +refuse. A sentence a run contradicts is reported and not used. + +## Limits + +- The parser reads simple conditional sentences in English. Anything else is + ignored unless you opt in to `--narration-model`. +- A condition must be a recorded field that became a run parameter, or text + on screen. "Patients over 65" and other computed conditions are not + supported. +- Two recordings that agree is weak evidence. Record one on each side of the + condition at least; more is better. +- A screen guard depends on OCR at run time. Prefer conditions on a recorded + field when you can say them that way. +- Branches on optional dialogs keep their own guard (the dialog being on + screen); narration is not applied to them. diff --git a/packages/flow/docs/PROBE_INDUCTION.md b/packages/flow/docs/PROBE_INDUCTION.md index 1ad8d1c74..33d4b0cd4 100644 --- a/packages/flow/docs/PROBE_INDUCTION.md +++ b/packages/flow/docs/PROBE_INDUCTION.md @@ -22,6 +22,7 @@ For each refused point, probing tries two readings: |---|---| | `incidental` | The recording without the step. | | `conditional` | The recording with the step, set to run only when its target (or its dialog) is on screen and to skip otherwise. This is the guard plain `induce` would put on the step. | +| `narrated` | Only when spoken narration proposed a guard here and no recording contradicts it: the recording with the step, set to run only when the narrated condition holds ([NARRATION.md](NARRATION.md)). | Each reading replays in every recording's context: that recording's own steps, values and subject, with only the point in question changed. Before @@ -44,6 +45,11 @@ exactly as without probing: - A read-back couldn't decide (store unreachable, effect uncertain, replay crashed). Such a run is logged as inconclusive and never counts as a miss. +The `narrated` reading has its own rule: it is certified when it produced the +recorded effect in every context and `incidental` conclusively failed in at +least one. The program then gets the narrated guard, and the sentence is kept +beside it. + Probing covers the points plain `induce` reports as `unconfirmed_optional` and `unconfirmed_branch`. Other refusals (contradicting paths, step order, loops with a consequential body) are logged as not probed and stay refused. @@ -148,9 +154,10 @@ refused, 3 when the target is refused. - One point at a time. While a point is probed, every other point keeps what each recording did. -- Only two readings per point. A step that should run under a condition other - than its own target being on screen (say, a field value) needs a recording, - as before. +- Two readings per point, three with narration. A step that should run under + a condition other than its own target being on screen (say, a field value) + needs a narrated sentence that names the condition, or a recording, as + before. - The check is only as good as the reward contract. A contract with no forbidden effects can't tell a harmless extra write from a harmful one. - The CI tests drive the real replayer against a scripted MockMed screen. The diff --git a/packages/flow/docs/commands/01-record-and-build/narration.md b/packages/flow/docs/commands/01-record-and-build/narration.md new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/openadapt_flow/__main__.py b/packages/flow/openadapt_flow/__main__.py index 36db4af05..9078f601c 100644 --- a/packages/flow/openadapt_flow/__main__.py +++ b/packages/flow/openadapt_flow/__main__.py @@ -1983,13 +1983,16 @@ def _cmd_induce(args: argparse.Namespace) -> int: else: traces.append(path) + spoken = _induce_narration_args(args) + if isinstance(spoken, int): + return spoken if args.probe_on: - probed = _induce_with_probes(args, traces) + probed = _induce_with_probes(args, traces, spoken) if isinstance(probed, int): return probed result = probed else: - result = induce_program(traces) + result = induce_program(traces, **spoken) print(result.render()) if args.report: report_path = Path(args.report) @@ -2023,7 +2026,53 @@ def _cmd_induce(args: argparse.Namespace) -> int: return 0 -def _induce_with_probes(args: argparse.Namespace, traces: list) -> Any: +def _induce_narration_args(args: argparse.Namespace) -> Any: + """Narration arguments for induce: each trace's narration.json, the + narrated guards a person accepted, and the opt-in compile-time model.""" + from openadapt_flow.compiler.narration_rules import ( + AnthropicNarrationInterpreter, + recording_screen_texts, + ) + from openadapt_flow.narration import NarrationRefused, load_narration + + narration = [] + for d in args.recording: + try: + narration.append(load_narration(Path(d))) + except (NarrationRefused, ValueError) as exc: + print(f"Narration not used for {d}: {exc}") + narration.append(None) + accepted = {} + if args.accept_narrated: + if not (args.accepted_by or "").strip(): + print("--accept-narrated needs --accepted-by NAME (who accepted it)") + return 2 + unpinned = [loc for loc in args.accept_narrated if "@" not in loc] + if unpinned: + print( + "--accept-narrated takes LOCATION@GUARD_ID as the report prints " + f"it (got {', '.join(unpinned)}), so it accepts that exact guard" + ) + return 2 + accepted = {loc: args.accepted_by.strip() for loc in args.accept_narrated} + frames = { + i: Path(d) + for i, d in enumerate(args.recording) + if (Path(d) / "frames").is_dir() + } + return { + "narration": narration, + "screen_texts": recording_screen_texts(frames) if frames else None, + "interpreter": AnthropicNarrationInterpreter() + if args.narration_model + else None, + "accepted_narration": accepted or None, + } + + +def _induce_with_probes( + args: argparse.Namespace, traces: list, spoken: Optional[dict] = None +) -> Any: """``induce --probe-on``: refuse a non-test target, else probe and induce. Returns the InductionResult, or an exit code when the target is refused. @@ -2070,7 +2119,11 @@ def _run(work: Path) -> Any: target = probing.DeploymentProbeTarget(cfg) print(f"Probing on declared test copy {cfg.name!r} (read-back: {contract})") return probing.probe_and_induce( - probe_traces, target=target, reward_contract=contract, work_dir=work + probe_traces, + target=target, + reward_contract=contract, + work_dir=work, + **(spoken or {}), ) if args.probe_work_dir: @@ -2081,6 +2134,55 @@ def _run(work: Path) -> Any: return _run(Path(tmp)) +def _cmd_narration_attach(args: argparse.Namespace) -> int: + import json + + from openadapt_flow.narration import ( + NarrationRefused, + attach_narration, + sentences_from_transcript, + transcribe_audio, + ) + + recording = Path(args.recording) + try: + from openadapt_flow.narration import require_scrubber + + scrubber = require_scrubber() # refuse before any transcription + if args.audio: + result = transcribe_audio( + args.audio, model=args.model, backend=args.backend + ) + else: + result = json.loads(Path(args.transcript).read_text(encoding="utf-8")) + path = attach_narration( + recording, + sentences_from_transcript(result, offset=args.offset), + scrubber=scrubber, + ) + except (NarrationRefused, FileNotFoundError, ValueError, ImportError) as exc: + print(f"Narration not written: {exc}") + return 2 + print(f"Scrubbed narration written to {path}") + return _cmd_narration_show(args) + + +def _cmd_narration_show(args: argparse.Namespace) -> int: + from openadapt_flow.narration import NarrationRefused, load_narration + + try: + narration = load_narration(Path(args.recording)) + except NarrationRefused as exc: + print(str(exc)) + return 2 + if narration is None: + print(f"No narration.json in {args.recording}") + return 1 + for line in narration.lines: + print(f"[{line.t_start:7.2f}s] {line.step_id or '-':>9} {line.text}") + return 0 + + def _cmd_for_each(args: argparse.Namespace) -> int: import shutil @@ -8023,8 +8125,74 @@ def build_parser() -> argparse.ArgumentParser: default=None, help="Write the induction report (JSON, every probe run included) here", ) + p.add_argument( + "--accept-narrated", + action="append", + default=[], + metavar="LOCATION@GUARD_ID", + help=( + "Accept the guard narration proposed at LOCATION, pinned by its " + "GUARD_ID, exactly as the report prints it (e.g. opt_2@3fa9c1d2e4b5). " + "A guard any recording contradicts stays refused. Repeatable; " + "needs --accepted-by" + ), + ) + p.add_argument( + "--accepted-by", + default=None, + help="Who accepted the narrated guard(s); recorded in the program", + ) + p.add_argument( + "--narration-model", + action="store_true", + help=( + "Let a compile-time model (your own Anthropic key) read narrated " + "sentences the built-in parser cannot. Its answer is a flagged " + "proposal, checked like any other; never used at run time. The " + "sentence is scrubbed before it is sent. See docs/NARRATION.md" + ), + ) p.set_defaults(func=_cmd_induce) + p = sub.add_parser( + "narration", + help=( + "Spoken narration for a recording: transcribe on this machine, " + "scrub, line up with the steps (narration.json), or show it" + ), + ) + nsub = p.add_subparsers(dest="narration_cmd", required=True) + pa = nsub.add_parser( + "attach", + help=( + "Transcribe audio on this machine (or read a transcript), scrub it " + "with openadapt-privacy, and line it up with the recording's steps" + ), + ) + pa.add_argument("recording", help="Recording directory (events.jsonl)") + src = pa.add_mutually_exclusive_group(required=True) + src.add_argument("--audio", help="16 kHz audio file (WAV, or FLAC with soundfile)") + src.add_argument( + "--transcript", + help="Whisper-style JSON transcript ({'segments': [{start, end, text}]})", + ) + pa.add_argument( + "--offset", + type=float, + default=0.0, + help="Seconds after the recording started that the audio starts (default 0)", + ) + pa.add_argument("--model", default="base", help="Whisper model (default: base)") + pa.add_argument( + "--backend", + default="auto", + help="On-device backend: auto | faster-whisper | openai-whisper", + ) + pa.set_defaults(func=_cmd_narration_attach) + ps = nsub.add_parser("show", help="Print a recording's narration by step") + ps.add_argument("recording", help="Recording or bundle directory") + ps.set_defaults(func=_cmd_narration_show) + p = sub.add_parser( "for-each", help=( diff --git a/packages/flow/openadapt_flow/adapters/capture.py b/packages/flow/openadapt_flow/adapters/capture.py index 42b370226..55d3d964a 100644 --- a/packages/flow/openadapt_flow/adapters/capture.py +++ b/packages/flow/openadapt_flow/adapters/capture.py @@ -1198,6 +1198,7 @@ def convert_capture( frames_dir.mkdir(parents=True, exist_ok=True) used_params: dict[str, str] = {} + step_times: list[dict[str, Any]] = [] viewport: Optional[list[int]] = None lines: list[str] = [] @@ -1244,6 +1245,7 @@ def convert_capture( line.update({k: v for k, v in event.items() if k != "_ts"}) line["t"] = round(ts - started_at, 3) lines.append(json.dumps(line)) + step_times.append({"i": i, "t": line["t"], "kind": event["kind"]}) (out_dir / "events.jsonl").write_text("\n".join(lines) + "\n") @@ -1293,6 +1295,42 @@ def convert_capture( elif desktop_provenance is not None: meta["desktop_capture"] = desktop_provenance (out_dir / "meta.json").write_text(json.dumps(meta, indent=2)) + _write_capture_narration(session, out_dir, started_at, step_times) return out_dir finally: session.close() + + +def _write_capture_narration( + session: Any, out_dir: Path, started_at: float, step_times: list[dict[str, Any]] +) -> None: + """Copy capture's scrubbed narration into the recording, lined up by time. + + Capture scrubbed the sentences before storing them (it refuses to store + narration otherwise) and its file says so; this only re-times them to the + recording's ``t`` clock and lines each one up with a step. A capture with + no narration writes nothing. + """ + reader = getattr(session, "narration", None) + if reader is None: + return + try: + utterances = reader() + except ValueError: + # Capture refuses a narration file that does not say it was scrubbed. + # The recording converts without narration rather than carrying it. + return + if not utterances: + return + from openadapt_flow.narration import Narration, align, write_narration + + sentences = [(u.start - started_at, u.end - started_at, u.text) for u in utterances] + write_narration( + out_dir, + Narration( + scrubbed=True, + scrubbed_by={"by": "openadapt-capture"}, + source="openadapt-capture", + lines=align(sentences, step_times), + ), + ) diff --git a/packages/flow/openadapt_flow/compiler/induction.py b/packages/flow/openadapt_flow/compiler/induction.py index dc6f30065..08c33ca17 100644 --- a/packages/flow/openadapt_flow/compiler/induction.py +++ b/packages/flow/openadapt_flow/compiler/induction.py @@ -94,7 +94,7 @@ import warnings from difflib import SequenceMatcher from pathlib import Path -from typing import Any, Optional, Protocol, Union, runtime_checkable +from typing import Any, Callable, Optional, Protocol, Sequence, Union, runtime_checkable from pydantic import BaseModel, Field @@ -107,6 +107,7 @@ from openadapt_flow.ir import ( ActionKind, Guard, + GuardSource, LoopSpec, ParamKind, ParamSpec, @@ -299,6 +300,43 @@ class ProbeVerdict(BaseModel): detail: str = "" +class NarratedGuard(BaseModel): + """A guard a spoken sentence suggested for one branch point. + + It starts as a flagged proposal. ``status`` is ``certified`` only when + every recording agrees with it (``certified_by="recordings_agree"``), + probing on a declared test copy confirmed it (``"probe"``), or a person + accepted it (``"person"``). A recording that contradicts it keeps the point + refused (``status="contradicted"``); a person cannot accept it then. + """ + + location: str = Field(description="opt_ or divergence_") + guard_id: str = Field( + default="", + description="Digest of the guard; accept it as @", + ) + sentence: str = Field(description="The scrubbed sentence") + recording: int = Field(description="Index of the recording that said it") + step_id: Optional[str] = None + guard: Predicate = Field(description="Run the step (or arm) when this holds") + rule: str = Field(description="The guard in plain words") + source: str = "narration" + status: str = Field(description="'certified' | 'proposed' | 'contradicted'") + certified_by: Optional[str] = None + accepted_by: Optional[str] = None + agree: list[int] = Field(default_factory=list) + contradict: list[int] = Field(default_factory=list) + unknown: list[int] = Field( + default_factory=list, + description="Recordings where the guard could not be checked", + ) + trace_params: dict[int, dict[str, str]] = Field( + default_factory=dict, + description="Per recording, the run parameter a param guard compares", + ) + detail: str = "" + + class InductionResult(BaseModel): """The induced program (or an honest refusal) plus the full audit trail.""" @@ -327,6 +365,10 @@ class InductionResult(BaseModel): "uncovered-branch notes on an emitted program (non-blocking)." ), ) + narrated_guards: list[NarratedGuard] = Field( + default_factory=list, + description="Guards spoken narration suggested, with how each was decided.", + ) probe_target: Optional[str] = Field( default=None, description="Name of the declared test copy probed (None: no probing).", @@ -362,6 +404,26 @@ def render(self) -> str: lines.append(f" [proposed] {p.target}: {p.content} (NOT trusted)") for u in self.uncertainties: lines.append(f" [REFUSED] {u.location}: {u.detail}") + for ng in self.narrated_guards: + if ng.status == "certified": + how = { + "recordings_agree": "every recording agrees", + "probe": "confirmed on the test copy", + "person": f"accepted by {ng.accepted_by}", + }.get(ng.certified_by or "", ng.certified_by or "") + verdict = f"CERTIFIED ({how})" + elif ng.status == "contradicted": + verdict = ( + "REFUSED (recording " + + ", ".join(str(t) for t in ng.contradict) + + " contradicts it)" + ) + else: + verdict = "proposed, NOT trusted" + lines.append( + f" [narrated] {ng.location}@{ng.guard_id}: {ng.sentence!r} -> " + f"{ng.rule} -- {verdict}" + ) if self.probe_target is not None: lines.append( f"probes on test copy {self.probe_target!r}: " @@ -975,6 +1037,10 @@ def induce_program( *, propose: Optional[Proposer] = None, probe_resolutions: Optional[dict[str, str]] = None, + narration: Optional[Sequence[Optional[Any]]] = None, + screen_texts: Optional[Callable[[int, str], Optional[Sequence[str]]]] = None, + interpreter: Optional[Any] = None, + accepted_narration: Optional[dict[str, str]] = None, ) -> InductionResult: """Induce a parameterized :class:`ProgramGraph` from multiple demonstrations of the same task -- the RFC §3 induction loop (step [4] + [5]). @@ -995,6 +1061,23 @@ def induce_program( is the only caller that sets it). ``"incidental"`` drops the step; ``"conditional"`` keeps it guarded on its own presence. A point not listed here is refused exactly as without probing. + ``"narrated@"`` certifies that narrated guard. + narration: One :class:`~openadapt_flow.narration.Narration` (or None) + per trace. Default: read ``narration.json`` from each trace that is + a recording or bundle directory. A sentence lined up near a branch + point PROPOSES a guard (:mod:`.narration_rules`); it is certified + only when every recording agrees, probing confirms it, or a person + accepts it (``accepted_narration``). + screen_texts: ``(trace, step_id) -> screen lines`` used to check a + screen-element guard. Default for recording directories: OCR of + the step's before-frame. + interpreter: Optional compile-time + :class:`~.narration_rules.NarrationInterpreter` for sentences the + parser cannot read. Its answer is checked like any narrated guard. + accepted_narration: ``{"@": person}`` -- + narrated guards a person accepted. The guard id pins the exact + guard the report showed; a different guard at that location is + not accepted. A guard a recording contradicts stays refused. Returns: An :class:`InductionResult` with the induced program (or a refusal) and @@ -1005,6 +1088,10 @@ def induce_program( result = InductionResult(n_traces=n) if n == 0: return result + if narration is None: + narration = _narration_from_traces(traces) + if screen_texts is None: + screen_texts = _default_screen_texts(traces) reduced = [_reduce_trace(wf.steps) for wf in workflows] columns = _align(reduced) @@ -1031,6 +1118,8 @@ def induce_program( data_sources: dict[str, Relation] = {} taken: set[str] = set() seen_div_groups: set[int] = set() + optional_nodes: dict[int, _Node] = {} + divergences: dict[int, tuple[list[_Column], int]] = {} for idx, col in enumerate(columns): present = sorted(col.tokens) @@ -1043,7 +1132,9 @@ def induce_program( seen_div_groups.add(grp) group_cols = [c for c in columns if c.divergent_group == grp] _handle_divergence(idx, group_cols, workflows, result, propose, decisions) - # Underdetermined divergence => refuse to emit (handled below). + # Underdetermined divergence => refuse to emit (handled below), + # unless narration settles it after the loop. + divergences[idx] = (group_cols, len(nodes)) continue # --- loop ---------------------------------------------------------- @@ -1072,8 +1163,27 @@ def induce_program( ) if node is not None: nodes.append(node) + optional_nodes[idx] = node decisions.append(dec) + guard_sources: dict[str, GuardSource] = {} + if narration and any(narration): + _apply_narration( + result=result, + columns=columns, + decisions=decisions, + nodes=nodes, + optional_nodes=optional_nodes, + divergences=divergences, + narration=list(narration), + n=n, + screen_texts=screen_texts, + interpreter=interpreter, + probe_resolutions=probe_resolutions or {}, + accepted=accepted_narration or {}, + guard_sources=guard_sources, + ) + result.column_decisions = decisions result.param_specs = param_specs @@ -1097,6 +1207,7 @@ def induce_program( for key, spec in param_specs.items() }, data_sources=data_sources, + guard_sources=guard_sources, ) result.program = program result.workflow = workflow @@ -1762,6 +1873,512 @@ def _maybe_propose( return None +# =========================================================================== +# Narration: a spoken sentence PROPOSES a guard; the recordings, a probe on a +# test copy, or a person certify it (see compiler.narration_rules). +# =========================================================================== + + +def _narration_from_traces(traces: list[TraceInput]) -> list[Optional[Any]]: + """``narration.json`` of each trace that is a directory (None otherwise). + + A file that does not say it was scrubbed is refused, which here means it + is not used. + """ + from openadapt_flow.narration import NarrationRefused, load_narration + + out: list[Optional[Any]] = [] + for trace in traces: + if isinstance(trace, (str, Path)) and Path(trace).is_dir(): + try: + out.append(load_narration(Path(trace))) + except (NarrationRefused, ValueError): + out.append(None) + else: + out.append(None) + return out + + +def _default_screen_texts(traces: list[TraceInput]): + from openadapt_flow.compiler.narration_rules import recording_screen_texts + + dirs = { + i: Path(t) + for i, t in enumerate(traces) + if isinstance(t, (str, Path)) and (Path(t) / "frames").is_dir() + } + return recording_screen_texts(dirs) if dirs else None + + +def _tok_steps(tok: Any) -> list[Step]: + if tok.kind == "single": + return [tok.step] + return [s for iteration in tok.iterations for s in iteration] + + +def _step_words(step: Step) -> set[str]: + from openadapt_flow.compiler.narration_rules import tokens + + parts = [step.intent or "", step.field_label or "", step.param or ""] + if step.anchor is not None: + parts.append(step.anchor.ocr_text or "") + return tokens(" ".join(parts)) + + +def _window_ids(columns: list[_Column], lo: int, hi: int, t: int) -> set[str]: + """Step ids of trace ``t`` around columns ``lo..hi``: its steps there plus + its nearest step before and after. A sentence lined up with one of these + was spoken at that point.""" + ids: set[str] = set() + for c in columns[lo : hi + 1]: + if t in c.tokens: + ids.update(s.id for s in _tok_steps(c.tokens[t])) + for c in reversed(columns[:lo]): + if t in c.tokens: + ids.add(_tok_steps(c.tokens[t])[-1].id) + break + for c in columns[hi + 1 :]: + if t in c.tokens: + ids.add(_tok_steps(c.tokens[t])[0].id) + break + return ids + + +def _next_step_id(columns: list[_Column], hi: int, t: int) -> Optional[str]: + for c in columns[hi + 1 :]: + if t in c.tokens: + return _tok_steps(c.tokens[t])[0].id + return None + + +def _param_fields(columns, decisions, n): + from openadapt_flow.compiler.narration_rules import FieldValues + + fields = [] + for dec in decisions: + if dec.kind != "param" or dec.param_name is None: + continue + col = columns[dec.index] + fields.append( + FieldValues( + param=dec.param_name, + label=dec.field, + values=[ + _value(col.tokens[t].step) + if t in col.tokens and col.tokens[t].kind == "single" + else None + for t in range(n) + ], + ) + ) + return fields + + +def _narrated_candidate(columns, lo, hi, narration, fields, descriptors, interpreter): + """The first sentence that reads as a rule about this point, or None. + + Sentences lined up at the point come first, then sentences anywhere whose + action names the step. With several arms (a divergence) the sentence must + be about exactly one of them. + """ + from openadapt_flow.compiler import narration_rules as nr + + ranked = [] + for t, nar in enumerate(narration): + if nar is None: + continue + window = _window_ids(columns, lo, hi, t) + for line in nar.lines: + ranked.append((0 if line.step_id in window else 1, t, line.t_start, line)) + ranked.sort(key=lambda r: r[:3]) + for rank, t, _, line in ranked: + rule = nr.parse_rule(line.text) + if rule is None: + continue + about = [ + k + for k, words in enumerate(descriptors) + if nr.is_about(rule, words, in_window=rank == 0) + ] + if len(about) != 1: + continue + cand = nr.candidate_from_rule( + rule, trace=t, step_id=line.step_id, fields=fields + ) + cand.target_arm = about[0] + return cand + if interpreter is None or len(descriptors) != 1: + return None + for rank, t, _, line in ranked: + if rank != 0 or nr.parse_rule(line.text) is not None: + continue + try: + data = interpreter.interpret( + line.text, + { + "fields": [f.param for f in fields], + "step": " ".join(sorted(descriptors[0])), + }, + ) + except Exception: # noqa: BLE001 - advisory path never breaks induction + data = None + if not data: + continue + answer = nr.candidate_from_model( + data, + sentence=line.text, + trace=t, + step_id=line.step_id, + fields=fields, + source=getattr(interpreter, "source", "narration+model"), + ) + if answer is not None: + answer.target_arm = 0 + return answer + return None + + +_HOW = { + "recordings_agree": "every recording agrees", + "probe": "confirmed on the test copy", + "person": "accepted by a person", +} + + +def _decide_narrated( + cand, + *, + location, + rule, + expected, + screen_step, + fields, + screen_texts, + probe_resolutions, + accepted, +) -> NarratedGuard: + from openadapt_flow.compiler.narration_rules import check_against_recordings + + agreement = check_against_recordings( + cand.guard, + expected, + fields=fields, + screen_step=screen_step, + screen_texts=screen_texts, + ) + certified_by = accepted_by = None + pinned = f"{location}@{cand.guard_id}" + if agreement.contradict: + status = "contradicted" + detail = ( + "recording(s) " + + ", ".join(str(t) for t in agreement.contradict) + + " did the opposite of what the sentence says" + ) + elif agreement.all_agree: + status, certified_by = "certified", "recordings_agree" + detail = f"all {len(agreement.agree)} recordings agree" + elif probe_resolutions.get(location) == f"narrated@{cand.guard_id}": + status, certified_by = "certified", "probe" + detail = "replaying it on the declared test copy produced every recorded effect" + elif accepted.get(pinned): + status, certified_by, accepted_by = "certified", "person", accepted[pinned] + detail = f"accepted by {accepted_by}" + else: + status = "proposed" + detail = ( + "could not be checked in recording(s) " + + ", ".join(str(t) for t in agreement.unknown) + if agreement.unknown + else "not checked" + ) + trace_params: dict[int, dict[str, str]] = {} + cond = cand.condition + if cond.kind is PredicateKind.PARAM_EQUALS: + f = next(x for x in fields if x.param == cond.param) + trace_params = { + t: {f.param: v} for t, v in enumerate(f.values) if v is not None + } + return NarratedGuard( + location=location, + guard_id=cand.guard_id, + sentence=cand.sentence, + recording=cand.trace, + step_id=cand.step_id, + guard=cand.guard, + rule=rule, + source=cand.source, + status=status, + certified_by=certified_by, + accepted_by=accepted_by, + agree=agreement.agree, + contradict=agreement.contradict, + unknown=agreement.unknown, + trace_params=trace_params, + detail=detail, + ) + + +def _guard_source(ng: NarratedGuard) -> GuardSource: + return GuardSource( + sentence=ng.sentence, + rule=ng.rule, + certified_by=ng.certified_by, # type: ignore[arg-type] + accepted_by=ng.accepted_by, + recording=ng.recording, + source=ng.source, # type: ignore[arg-type] + ) + + +def _flag_on_uncertainty(result, uncertainty_location, ng, proposal) -> None: + for u in result.uncertainties: + if u.location != uncertainty_location: + continue + u.proposal = proposal + if ng.status == "contradicted": + u.detail += ( + f" Narration suggested {ng.rule!r}, but {ng.detail}, so it is not used." + ) + continue + u.detail += f" Narration suggests: {ng.rule} ({ng.detail})." + u.record_next = ((u.record_next + " ") if u.record_next else "") + ( + "Or accept the narrated rule: induce --accept-narrated " + f"{ng.location}@{ng.guard_id} --accepted-by NAME." + ) + if u.question is not None and not any( + o.key == "narrated" for o in u.question.options + ): + u.question.options.append( + QuestionOption( + key="narrated", + label=f"Use the narrated rule: {ng.rule}", + effect=OptionEffect.NONE, + ) + ) + + +def _divergence_arms(group_cols, n): + """The two paths a divergence takes: (steps per arm, traces per arm). + + None unless there are exactly two, each the same steps with the same + values in every recording that took it. + """ + arms: dict[tuple, list[tuple[int, list[Step]]]] = {} + for t in range(n): + steps: list[Step] = [] + for c in group_cols: + tok = c.tokens.get(t) + if tok is None: + continue + if tok.kind != "single": + return None + steps.append(tok.step) + arms.setdefault(tuple(_sig(s) for s in steps), []).append((t, steps)) + if len(arms) != 2: + return None + arm_steps, members = [], [] + for entries in arms.values(): + first = entries[0][1] + for _, steps in entries[1:]: + if [_value(s) for s in steps] != [_value(s) for s in first]: + return None + arm_steps.append(first) + members.append([t for t, _ in entries]) + return arm_steps, members + + +def _branch_node(idx: int, guard: Predicate, arm_steps, target: int) -> _Node: + bid = f"branch_div_{idx}" + node = _Node(bid) + entries: list[Optional[str]] = [] + for k, steps in enumerate(arm_steps): + ids = [f"div{idx}_{k}_{j}" for j in range(len(steps))] + for j, step in enumerate(steps): + tr = Transition(target=ids[j + 1] if j + 1 < len(ids) else "__PATCH__") + node.states[ids[j]] = State( + id=ids[j], + kind=StateKind.ACTION, + step=step.model_copy(deep=True), + transitions=[tr], + ) + if j + 1 == len(ids): + node.exits.append(tr) + entries.append(ids[0] if ids else None) + other = 1 - target + t_rule = Transition( + guard=guard, target=entries[target] or "__PATCH__", label="narrated rule holds" + ) + t_else = Transition(target=entries[other] or "__PATCH__", label="otherwise") + for tr, entry in ((t_rule, entries[target]), (t_else, entries[other])): + if entry is None: + node.exits.append(tr) + node.states[bid] = State( + id=bid, kind=StateKind.BRANCH, transitions=[t_rule, t_else] + ) + return node + + +def _apply_narration( + *, + result, + columns, + decisions, + nodes, + optional_nodes, + divergences, + narration, + n, + screen_texts, + interpreter, + probe_resolutions, + accepted, + guard_sources, +) -> None: + fields = _param_fields(columns, decisions, n) + + # -- optional steps (present in some recordings, absent in others) ------ + for idx, node in optional_nodes.items(): + col = columns[idx] + present = sorted(col.tokens) + step0 = col.tokens[present[0]].step + sid = f"opt_{idx}" + if _looks_like_dialog(step0) or probe_resolutions.get(sid) in ( + "incidental", + "conditional", + ): + continue + state = node.states.get(sid) + if state is None or state.step is None: + continue + cand = _narrated_candidate( + columns, idx, idx, narration, fields, [_step_words(step0)], interpreter + ) + if cand is None: + continue + field = _field_key(step0) + ng = _decide_narrated( + cand, + location=sid, + rule=cand.describe(f"run {step0.intent!r}"), + expected={t: t in col.tokens for t in range(n)}, + screen_step={ + t: col.tokens[t].step.id + if t in col.tokens + else _next_step_id(columns, idx, t) + for t in range(n) + }, + fields=fields, + screen_texts=screen_texts, + probe_resolutions=probe_resolutions, + accepted=accepted, + ) + result.narrated_guards.append(ng) + if ng.status == "certified": + state.step.guard = Guard(predicate=cand.guard, on_unmet="skip") + result.uncertainties = [ + u for u in result.uncertainties if u.location != sid + ] + result.inferred = [ + line + for line in result.inferred + if not line.startswith(f"optional step '{field}' at column {idx} ") + ] + result.inferred.append( + f"guard at {sid}: {ng.rule} -- suggested by narration, " + f"certified: {_HOW.get(ng.certified_by or '', '')}" + ) + guard_sources[sid] = _guard_source(ng) + for dec in decisions: + if dec.index == idx: + dec.kind = "branch" + dec.note = f"narrated guard on '{field}' ({ng.certified_by})" + continue + proposal = Proposal( + target=sid, + kind="guard", + content=f"{ng.rule} (narration: {ng.sentence!r})", + source=ng.source, + ) + result.proposed.append(proposal) + _flag_on_uncertainty(result, sid, ng, proposal) + + # -- divergences (recordings took different paths at the same point) ---- + # Insert from the back so earlier positions stay valid; at equal positions + # the later column goes in first, so the earlier one ends up in front. + for idx, (group_cols, pos) in sorted( + divergences.items(), key=lambda kv: (-kv[1][1], -kv[0]) + ): + where = f"column {idx}" + location = f"divergence_{idx}" + if not any( + u.location == where and u.kind == "ambiguous_branch" + for u in result.uncertainties + ): + continue + arms = _divergence_arms(group_cols, n) + if arms is None: + continue + arm_steps, members = arms + indices = [i for i, c in enumerate(columns) if any(c is g for g in group_cols)] + lo, hi = min(indices), max(indices) + if indices != list(range(lo, hi + 1)): + # Another column sits inside the divergence; one branch node at + # its first position would reorder steps. Leave it refused. + continue + descriptors = [ + set().union(*(_step_words(s) for s in steps)) if steps else set() + for steps in arm_steps + ] + cand = _narrated_candidate( + columns, lo, hi, narration, fields, descriptors, interpreter + ) + if cand is None: + continue + target = cand.target_arm or 0 + label = " then ".join(s.intent for s in arm_steps[target]) or "empty" + screen_step = {} + for t in members[0] + members[1]: + first = next( + (c.tokens[t].step.id for c in group_cols if t in c.tokens), None + ) + screen_step[t] = first or _next_step_id(columns, hi, t) + ng = _decide_narrated( + cand, + location=location, + rule=cand.describe(f"take the path {label!r}"), + expected={t: t in members[target] for t in members[0] + members[1]}, + screen_step=screen_step, + fields=fields, + screen_texts=screen_texts, + probe_resolutions=probe_resolutions, + accepted=accepted, + ) + result.narrated_guards.append(ng) + if ng.status == "certified": + nodes.insert(pos, _branch_node(idx, cand.guard, arm_steps, target)) + result.uncertainties = [ + u for u in result.uncertainties if u.location != where + ] + result.inferred.append( + f"branch at {location}: {ng.rule} -- suggested by narration, " + f"certified: {_HOW.get(ng.certified_by or '', '')}" + ) + guard_sources[f"branch_div_{idx}"] = _guard_source(ng) + for dec in decisions: + if dec.index == idx: + dec.kind = "branch" + dec.note = f"narrated branch ({ng.certified_by})" + continue + proposal = Proposal( + target=location, + kind="guard", + content=f"{ng.rule} (narration: {ng.sentence!r})", + source=ng.source, + ) + result.proposed.append(proposal) + _flag_on_uncertainty(result, where, ng, proposal) + + def _wire(nodes: list[_Node]): """Chain nodes into a single program graph + collect loop-body subflows.""" if not nodes: diff --git a/packages/flow/openadapt_flow/compiler/narration_rules.py b/packages/flow/openadapt_flow/compiler/narration_rules.py new file mode 100644 index 000000000..1599aa9d3 --- /dev/null +++ b/packages/flow/openadapt_flow/compiler/narration_rules.py @@ -0,0 +1,590 @@ +"""Turn a spoken sentence into a proposed branch guard, and check it. + +When ``induce`` (or ``teach-branch``) finds a branch, the recordings show +*that* the person did something different, not *why*. Narration often says +why: "if the insurance is Medicare, skip the copay screen". This module + +1. parses such a sentence into a condition and an action + (:func:`parse_rule`, deterministic; an optional compile-time + :class:`NarrationInterpreter` may help on the customer's own key), +2. maps the condition onto something the runtime can check without a model: + a recorded field (``param_equals`` on a run parameter) or a screen element + (``text_present``), and +3. checks the proposed guard against every recording + (:func:`check_against_recordings`). + +A narrated guard is a **proposal**. It becomes a certified guard only when +every recording agrees with it, when ``induce --probe-on`` confirms it on a +test copy, or when a person accepts it. A recording that contradicts it keeps +the point refused. At run time the guard is an ordinary predicate: a string +compare or an OCR presence check, never a model call. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import Any, Callable, Optional, Protocol, Sequence, runtime_checkable + +from openadapt_flow.ir import Predicate, PredicateKind + +_WORD = re.compile(r"[a-z0-9]+") +_STOP = frozenset( + """a an the is are was be to of on in at for and or but so then i we you it + its it's this that they them their my our your just can will should would + do does did dont don't not no need have has had go going now here there + screen page step field box button tab form when if since because unless + as with by from into out up down please also skip skipping skipped ignore + leave omit bypass never always only don t s""".split() +) +_SKIP_VERBS = ( + "skip", + "don't", + "dont", + "do not", + "no need to", + "ignore", + "leave out", + "bypass", + "omit", + "never", + "won't", + "will not", +) +_MARKERS = re.compile( + r"\b(if|when|whenever|unless|since|because|as long as|only if|only when)\b", + re.IGNORECASE, +) +_COPULA = re.compile( + r"^(?P.*?)\s*\b(?Pis not|isn't|isnt|is|are not|aren't|are|was|" + r"equals|says|shows|reads|=)\s+(?P.+)$", + re.IGNORECASE, +) +_PRONOUN_SUBJECTS = frozenset( + {"", "it", "this", "that", "they", "he", "she", "its", "it's"} +) + + +@dataclass(frozen=True) +class ParsedRule: + """A sentence split into a condition and an action.""" + + sentence: str + value: str + subject: Optional[str] + negated: bool + action: str + skip: bool + + @property + def runs_when_condition_holds(self) -> bool: + """True: the action happens when the condition holds. A "skip" (or an + "unless", or "is not") flips it.""" + return self.skip == self.negated + + +def tokens(text: str) -> set[str]: + """Content words, lower-case, with a trailing plural ``s`` dropped.""" + out = set() + for w in _WORD.findall((text or "").lower().replace("_", " ")): + if w in _STOP or len(w) < 2: + continue + out.add(w[:-1] if len(w) > 3 and w.endswith("s") else w) + return out + + +def _norm_value(text: str) -> str: + text = re.sub(r"[\"'“”‘’]", "", text or "").strip().lower() + text = re.sub(r"^(a|an|the)\s+", "", text) + return re.sub(r"\s+", " ", text).strip(" .,;:!?") + + +def _clean_value(text: str) -> str: + text = re.sub(r"[\"“”]", "", text or "").strip() + text = re.sub(r"^(a|an|the)\s+", "", text, flags=re.IGNORECASE) + # "Medicare, so" / "Medicare then" -> "Medicare" + text = re.split(r"\s*(?:,|;|\bthen\b|\bso\b)\s*", text, maxsplit=1)[0] + return text.strip(" .,;:!?") + + +_OTHERWISE = re.compile(r",?\s*\b(?:otherwise|or else|else)\b", re.I) +_UNCLEAR_NEGATION = re.compile( + r"\b(?:not|no|never|n't|isn't|aren't|won't|don't)\b", re.I +) +_NOT_NEEDED = re.compile( + r"^(?P.+?)\s+(?:is not|isn't|are not|aren't)\s+(?:needed|required|necessary)$" + r"|^(?P.+?)\s+(?:is|are)\s+(?:not needed|not required|unnecessary)$", + re.I, +) + + +def _split_action(action: str) -> tuple[str, Optional[bool]]: + """``(object, skip)``; ``skip`` is None when the negation is unclear.""" + action = _OTHERWISE.split(action, maxsplit=1)[0] + a = re.sub(r"^(?:then|so|and)\s+", "", action.strip(" ,.;:!?"), flags=re.I) + a = re.sub( + r"^(?:i|we|you)\s+(?:can\s+|will\s+|should\s+|just\s+|always\s+|need to\s+)*", + "", + a, + flags=re.I, + ).strip() + low = a.lower() + blank = re.match(r"^leave\s+(?P.+?)\s+(?:blank|empty|alone)$", a, flags=re.I) + if blank is not None: + return blank.group("obj"), True + needless = _NOT_NEEDED.match(a) + if needless is not None: + return needless.group("obj") or needless.group("obj2"), True + for verb in _SKIP_VERBS: + if low.startswith(verb + " ") or low == verb: + rest = a[len(verb) :].strip() + rest = re.sub( + r"^(?:need to|have to|bother|fill in|do)\s+", "", rest, flags=re.I + ) + # "don't skip X": a double negative is too easy to misread. + if any(rest.lower().startswith(v + " ") for v in _SKIP_VERBS): + return rest, None + if _UNCLEAR_NEGATION.search(rest): + return rest, None + return rest, True + if _UNCLEAR_NEGATION.search(a): + return a, None + return a, False + + +def parse_rule(sentence: str) -> Optional[ParsedRule]: + """Split a narrated sentence into a condition and an action, or None. + + Handles the shapes people use while demonstrating: + + * "If the insurance is Medicare, skip the copay screen." + * "Skip the copay screen when the insurance is Medicare." + * "Since it's Medicare, I don't fill in the copay." + * "The insurance is Medicare, so I skip copay." + * "Unless the payer is Aetna, leave the referral blank." + """ + text = re.sub(r"\s+", " ", (sentence or "").strip()) + if not text: + return None + marker = _MARKERS.search(text) + negated = False + if marker is not None: + word = marker.group(1).lower() + negated = word == "unless" + after = text[marker.end() :] + before = text[: marker.start()].strip(" ,.;:") + boundary = re.search(r",|;|\bthen\b|\bso\b", after) + if boundary is not None: + condition = after[: boundary.start()] + action = after[boundary.end() :] + else: + condition, action = after, "" + # "..., otherwise enter it": only the first action is the rule. + action = _OTHERWISE.split(action, maxsplit=1)[0] + if not action.strip(" .,;:!?"): + action = before + elif before and not tokens(action): + action = before + else: + so = re.search(r",?\s*\bso\b\s+", text) + if so is None: + return None + condition, action = text[: so.start()], text[so.end() :] + condition = condition.strip(" ,.;:!?") + match = _COPULA.match(condition) + if match is not None: + subject = match.group("subj").strip() + subject = re.sub(r"^(?:the|a|an|its|their)\s+", "", subject, flags=re.I) + value = _clean_value(match.group("val")) + cop = match.group("cop").lower() + if cop in ("is not", "isn't", "isnt", "are not", "aren't"): + negated = not negated + else: + lowered = condition.lower() + subject = "" + for pron in ("it's ", "it is ", "this is ", "they're ", "they are "): + if lowered.startswith(pron): + condition = condition[len(pron) :] + break + flipped = re.match(r"^(?:not|n't)\s+", condition, flags=re.I) + if flipped is not None: + condition = condition[flipped.end() :] + negated = not negated + value = _clean_value(condition) + if re.match(r"^(?:not|no)\b", value, flags=re.I): + return None # a negation the parser did not place + if subject.lower() in _PRONOUN_SUBJECTS: + subject = "" + action, skip = _split_action(action) + if skip is None or not value or not action and not skip: + return None + return ParsedRule( + sentence=text, + value=value, + subject=subject or None, + negated=negated, + action=action, + skip=skip, + ) + + +# -- the compile-time model, behind an interface ------------------------------ + + +@runtime_checkable +class NarrationInterpreter(Protocol): + """Turns a sentence the parser could not read into a condition. + + Called at COMPILE time only, at most once per branch point. Returns + ``{"field": | None, "text": | + None, "value": str, "step_runs_when_true": bool}`` or None. Whatever it + returns is checked against the recordings like any narrated guard, and it + may only name a field it was offered. + """ + + source: str + + def interpret( + self, sentence: str, context: dict[str, Any] + ) -> Optional[dict[str, Any]]: ... + + +class AnthropicNarrationInterpreter: + """Compile-time interpreter on the customer's own Anthropic key. + + Imports the SDK lazily; constructing it needs no key. The sentence (already + scrubbed) is scrubbed again on the way out, or the call is refused when + openadapt-privacy is missing (``openadapt_flow.privacy.prepare_outbound``). + Never used at run time. + """ + + source = "narration+model" + + def __init__(self, *, client: Any = None, model: Optional[str] = None) -> None: + self._client = client + self._model = model + + def interpret( + self, sentence: str, context: dict[str, Any] + ) -> Optional[dict[str, Any]]: + import json + + from openadapt_flow import privacy + + payload = privacy.prepare_outbound( + {"sentence": sentence, **context}, "narration rule interpretation" + ) + client = self._client + if client is None: + import anthropic + + from openadapt_flow.benchmark.agent_baseline import load_api_key + + client = anthropic.Anthropic(api_key=load_api_key()) + model = self._model + if model is None: + from openadapt_flow.benchmark.agent_baseline import MODEL + + model = MODEL + prompt = ( + "A person demonstrating a workflow said the sentence below. Say which " + "condition decides whether the step runs. Use only a field from " + "'fields', or text that would be on screen. Reply with JSON only: " + '{"field": str|null, "text": str|null, "value": str, ' + '"step_runs_when_true": bool}. Reply null if the sentence names no ' + "condition.\n" + json.dumps(payload) + ) + response = client.messages.create( + model=model, max_tokens=200, messages=[{"role": "user", "content": prompt}] + ) + raw = "".join( + getattr(b, "text", "") + for b in response.content + if getattr(b, "type", "") == "text" + ).strip() + try: + data = json.loads(raw) + except ValueError: + return None + return data if isinstance(data, dict) else None + + +# -- mapping a rule onto a checkable guard ------------------------------------ + + +@dataclass +class FieldValues: + """A recorded field that became a run parameter, with each recording's value.""" + + param: str + label: str + values: list[Optional[str]] + + +@dataclass +class Candidate: + """A guard proposed from one sentence, before it is checked.""" + + sentence: str + trace: int + step_id: Optional[str] + condition: Predicate + runs_when_condition_holds: bool + source: str = "narration" + target_arm: Optional[int] = None # divergence: the arm the sentence is about + + @property + def guard_id(self) -> str: + """Short digest of the guard, so an acceptance or a probe result pins + this exact guard and not whatever a later pass proposes.""" + from openadapt_flow.ir import predicate_contract_sha256 + + return predicate_contract_sha256(self.guard)[:12] + + @property + def guard(self) -> Predicate: + """The predicate under which the step (or target arm) runs.""" + if self.runs_when_condition_holds: + return self.condition + return Predicate( + kind=PredicateKind.NOT, + operands=[self.condition], + intent=f"not ({self.condition.intent})", + ) + + def describe(self, what: str) -> str: + cond = self.condition + if cond.kind is PredicateKind.PARAM_EQUALS: + fact = f"{cond.param} is {cond.value!r}" + nfact = f"{cond.param} is not {cond.value!r}" + else: + fact = f"the screen shows {cond.text!r}" + nfact = f"the screen does not show {cond.text!r}" + return f"{what} only when {fact if self.runs_when_condition_holds else nfact}" + + +def _match_field( + subject: Optional[str], value: str, fields: Sequence[FieldValues] +) -> Optional[tuple[FieldValues, str]]: + want = _norm_value(value) + with_value = [ + (f, v) + for f in fields + for v in dict.fromkeys(x for x in f.values if x is not None) + if _norm_value(v) == want + ] + if subject: + subj = tokens(subject) + named = [ + (f, v) for f, v in with_value if subj & (tokens(f.param) | tokens(f.label)) + ] + if len({f.param for f, _ in named}) == 1: + return named[0] + if named: + return None + if len({f.param for f, _ in with_value}) == 1: + return with_value[0] + return None + + +def candidate_from_rule( + rule: ParsedRule, + *, + trace: int, + step_id: Optional[str], + fields: Sequence[FieldValues], +) -> Candidate: + """Map a parsed rule onto a recorded field when one matches, else the screen.""" + matched = _match_field(rule.subject, rule.value, fields) + if matched is not None: + f, recorded = matched + condition = Predicate( + kind=PredicateKind.PARAM_EQUALS, + param=f.param, + value=recorded, + intent=f"{f.param} is {recorded}", + ) + else: + condition = Predicate( + kind=PredicateKind.TEXT_PRESENT, + text=rule.value, + intent=f"{rule.value} on screen", + ) + return Candidate( + sentence=rule.sentence, + trace=trace, + step_id=step_id, + condition=condition, + runs_when_condition_holds=rule.runs_when_condition_holds, + ) + + +def candidate_from_model( + data: dict[str, Any], + *, + sentence: str, + trace: int, + step_id: Optional[str], + fields: Sequence[FieldValues], + source: str, +) -> Optional[Candidate]: + """Validate an interpreter's answer. It may only name an offered field and + a value some recording actually had; anything else is dropped.""" + value = str(data.get("value") or "").strip() + runs = data.get("step_runs_when_true") + if not value or not isinstance(runs, bool): + return None # only a real JSON boolean says which way the guard goes + name = data.get("field") + if name: + f = next((x for x in fields if x.param == name), None) + if f is None: + return None + recorded = next( + ( + v + for v in f.values + if v is not None and _norm_value(v) == _norm_value(value) + ), + None, + ) + if recorded is None: + return None + condition = Predicate( + kind=PredicateKind.PARAM_EQUALS, + param=f.param, + value=recorded, + intent=f"{f.param} is {recorded}", + ) + else: + text = str(data.get("text") or value).strip() + # The screen text must be words the person actually said. + if not text or _norm_value(text) not in _norm_value(sentence): + return None + condition = Predicate( + kind=PredicateKind.TEXT_PRESENT, text=text, intent=f"{text} on screen" + ) + return Candidate( + sentence=sentence, + trace=trace, + step_id=step_id, + condition=condition, + runs_when_condition_holds=runs, + source=source, + ) + + +def is_about(rule: ParsedRule, descriptor: set[str], *, in_window: bool) -> bool: + """Does the rule's action talk about this step (or arm)? + + An action with content words must share one with the step. An action with + none ("skip it") counts only when the sentence was spoken right there. + """ + words = tokens(rule.action) + if not words: + return in_window + return bool(words & descriptor) + + +# -- checking a guard against the recordings ---------------------------------- + +ScreenTexts = Callable[[int, str], Optional[Sequence[str]]] + + +@dataclass +class Agreement: + agree: list[int] = field(default_factory=list) + contradict: list[int] = field(default_factory=list) + unknown: list[int] = field(default_factory=list) + + @property + def all_agree(self) -> bool: + return bool(self.agree) and not self.contradict and not self.unknown + + +def _holds( + pred: Predicate, + trace: int, + fields: Sequence[FieldValues], + screen: Optional[Sequence[str]], +) -> Optional[bool]: + kind = pred.kind + if kind is PredicateKind.NOT: + inner = _holds(pred.operands[0], trace, fields, screen) + return None if inner is None else not inner + if kind is PredicateKind.PARAM_EQUALS: + f = next((x for x in fields if x.param == pred.param), None) + if f is None or trace >= len(f.values) or f.values[trace] is None: + return None + # The run-time check is an exact string compare + # (runtime.program_predicates); check the recordings the same way, so + # "medicare" and "Medicare" are different values here too. + return f.values[trace] == pred.value + if kind is PredicateKind.TEXT_PRESENT: + if screen is None: + return None + want = _norm_value(pred.text or "") + return any(want and want in _norm_value(line) for line in screen) + return None + + +def check_against_recordings( + guard: Predicate, + expected: dict[int, bool], + *, + fields: Sequence[FieldValues], + screen_step: dict[int, Optional[str]], + screen_texts: Optional[ScreenTexts] = None, +) -> Agreement: + """Evaluate ``guard`` in every recording and compare with what it did. + + ``expected[t]`` is True when recording ``t`` took the guarded path. + ``screen_step[t]`` names the step whose before-screen is where the guard + would be checked in recording ``t`` (for screen-element guards). + """ + out = Agreement() + for t, took in sorted(expected.items()): + screen = None + step_id = screen_step.get(t) + if screen_texts is not None and step_id is not None: + screen = screen_texts(t, step_id) + holds = _holds(guard, t, fields, screen) + if holds is None: + out.unknown.append(t) + elif holds == took: + out.agree.append(t) + else: + out.contradict.append(t) + return out + + +def recording_screen_texts(recording_dirs: dict[int, Any]) -> ScreenTexts: + """Screen text at a step, read by OCR from a recording's before-frame. + + ``recording_dirs`` maps trace index -> recording directory. Steps are + ``step_``. Returns None when the frame or OCR is missing, so + the guard stays unchecked rather than guessed. + """ + from pathlib import Path + + cache: dict[tuple[int, str], Optional[list[str]]] = {} + + def read(trace: int, step_id: str) -> Optional[Sequence[str]]: + key = (trace, step_id) + if key in cache: + return cache[key] + texts: Optional[list[str]] = None + root = recording_dirs.get(trace) + m = re.fullmatch(r"step_(\d+)", step_id or "") + if root is not None and m: + frame = Path(root) / "frames" / f"{int(m.group(1)):04d}_before.png" + if frame.is_file(): + try: + from openadapt_flow.vision.ocr import ocr + + texts = [line.text for line in ocr(frame.read_bytes())] + except Exception: # noqa: BLE001 - no OCR means unchecked + texts = None + cache[key] = texts + return texts + + return read diff --git a/packages/flow/openadapt_flow/compiler/probing.py b/packages/flow/openadapt_flow/compiler/probing.py index bcef2d9fd..bfe68b08f 100644 --- a/packages/flow/openadapt_flow/compiler/probing.py +++ b/packages/flow/openadapt_flow/compiler/probing.py @@ -12,6 +12,11 @@ * ``incidental``: the step is noise. Every recording runs without it. * ``conditional``: the step is a required branch. Every recording runs it, guarded to skip when its target (or its dialog) is not on screen. +* ``narrated`` (only when narration proposed a guard there and no recording + contradicts it): every recording runs the step guarded by the narrated + condition (:mod:`openadapt_flow.compiler.narration_rules`). It is confirmed + when it produces the recorded effect in every context and ``incidental`` + conclusively fails in at least one. Each run is scored by reading the system of record back through the reward contract's oracle and judging it with @@ -46,6 +51,7 @@ from openadapt_flow.compiler.induction import ( InductionResult, + NarratedGuard, OptionalPoint, ProbeRun, ProbeVerdict, @@ -55,9 +61,10 @@ optional_guard, optional_points, ) -from openadapt_flow.ir import Step, Workflow +from openadapt_flow.ir import Guard, Step, Workflow INTERPRETATIONS: tuple[str, ...] = ("incidental", "conditional") +NARRATED = "narrated" #: Runtime signals the serve-reward judge accepts. RUNTIME_SIGNALS = frozenset( @@ -180,14 +187,18 @@ def require_test_copy(cfg: Any) -> None: # --------------------------------------------------------------------------- -def _guarded_copy(step: Step) -> Step: +def _guarded_copy(step: Step, guard: Optional[Guard] = None) -> Step: copy = step.model_copy(deep=True) - copy.guard = optional_guard(step) + copy.guard = guard if guard is not None else optional_guard(step) return copy def interpretation_workflow( - traces: list[ProbeTrace], point: OptionalPoint, t: int, interpretation: str + traces: list[ProbeTrace], + point: OptionalPoint, + t: int, + interpretation: str, + narrated: Optional[NarratedGuard] = None, ) -> tuple[Workflow, Optional[int]]: """Recording ``t`` replayed under ``interpretation`` of ``point``. @@ -203,13 +214,18 @@ def interpretation_workflow( if interpretation == "incidental": if t in point.index_in_trace: del steps[point.index_in_trace[t]] - elif interpretation == "conditional": + elif interpretation in ("conditional", NARRATED): + guard = None + if interpretation == NARRATED: + if narrated is None: + raise ValueError("the narrated interpretation needs a narrated guard") + guard = Guard(predicate=narrated.guard, on_unmet="skip") if t in point.index_in_trace: k = point.index_in_trace[t] - steps[k] = _guarded_copy(wf.steps[k]) + steps[k] = _guarded_copy(wf.steps[k], guard) else: inserted = point.insert_at[t] - steps.insert(inserted, _guarded_copy(point.step)) + steps.insert(inserted, _guarded_copy(point.step, guard)) else: raise ValueError(f"unknown interpretation {interpretation!r}") out = wf.model_copy(update={"steps": steps, "program": None}) @@ -310,9 +326,17 @@ def _probe_once( interpretation: str, identity: dict[str, str], work_dir: Path, + narrated: Optional[NarratedGuard] = None, ) -> ProbeRun: slug = f"{point.location}-{interpretation}-{t}" - workflow, inserted = interpretation_workflow(traces, point, t, interpretation) + workflow, inserted = interpretation_workflow( + traces, point, t, interpretation, narrated + ) + params = _params(traces[t]) + if interpretation == NARRATED and narrated is not None: + # A guard on a recorded field compares the run parameter induction + # names; give each run its recording's value for it. + params.update(narrated.trace_params.get(t, {})) base: dict[str, Any] = { "location": point.location, "interpretation": interpretation, @@ -338,7 +362,7 @@ def _probe_once( workflow, bundle_dir=work_dir / "bundles" / slug, run_dir=work_dir / "runs" / slug, - params=_params(traces[t]), + params=params, ) except Exception as exc: # noqa: BLE001 - a crashed replay is a platform failure signal = RunSignal("failed_platform", detail=f"replay raised: {exc}") @@ -373,9 +397,11 @@ def _probe_point( point: OptionalPoint, work_dir: Path, runs: list[ProbeRun], + narrated: Optional[NarratedGuard] = None, ) -> ProbeVerdict: status: dict[str, str] = {} - for interpretation in INTERPRETATIONS: + readings = INTERPRETATIONS + ((NARRATED,) if narrated is not None else ()) + for interpretation in readings: state = "passed" for t in range(len(traces)): run = _probe_once( @@ -387,6 +413,7 @@ def _probe_point( interpretation=interpretation, identity=identities[t], work_dir=work_dir, + narrated=narrated, ) runs.append(run) if run.produced_recorded_effect: @@ -396,8 +423,22 @@ def _probe_point( break # one conclusive miss rules this interpretation out state = "inconclusive" # keep looking for a conclusive miss status[interpretation] = state - passed = [k for k, v in status.items() if v == "passed"] - others_failed = all(v == "failed" for k, v in status.items() if k not in passed) + if narrated is not None: + if status[NARRATED] == "passed" and status["incidental"] == "failed": + return ProbeVerdict( + location=point.location, + kind=point.kind, + interpretations=status, + certified=NARRATED, + detail=( + f"the narrated guard ({narrated.rule}) produced the recorded " + f"effect in all {len(traces)} recording contexts and dropping " + "the step did not" + ), + ) + base = {k: v for k, v in status.items() if k in INTERPRETATIONS} + passed = [k for k, v in base.items() if v == "passed"] + others_failed = all(v == "failed" for k, v in base.items() if k not in passed) if len(passed) == 1 and others_failed: return ProbeVerdict( location=point.location, @@ -411,7 +452,7 @@ def _probe_point( ) if len(passed) > 1: detail = "more than one interpretation produced the recorded effect" - elif not passed and all(v == "failed" for v in status.values()): + elif not passed and all(v == "failed" for v in base.values()): detail = "no interpretation produced the recorded effect" else: detail = "a read-back could not decide" @@ -430,9 +471,17 @@ def probe_and_induce( reward_contract: Union[Any, str, Path], work_dir: Path, propose: Optional[Proposer] = None, + narration: Optional[list[Any]] = None, + screen_texts: Any = None, + interpreter: Any = None, + accepted_narration: Optional[dict[str, str]] = None, ) -> InductionResult: """Induce, then probe each refused optional point on ``target``. + ``narration`` (one per trace) and the other narration arguments are passed + to :func:`induce_program`. Where narration proposed a guard that no + recording contradicts, the ``narrated`` reading is probed as well. + ``target`` must already have passed :func:`require_test_copy` (the CLI does this from the deployment file). ``reward_contract`` is a ``serve-reward`` contract bundle (a directory or a loaded @@ -453,10 +502,18 @@ def probe_and_induce( workflows: list[TraceInput] = [tr.workflow for tr in traces] # The proposer is advisory and never decides a probe, so the first pass # runs without it; it is consulted once, in the pass whose result returns. - first = induce_program(workflows) + spoken: dict[str, Any] = { + "narration": narration if narration is not None else [None] * len(traces), + "screen_texts": screen_texts, + "interpreter": interpreter, + "accepted_narration": accepted_narration, + } + first = induce_program(workflows, **spoken) name = str(getattr(target, "name", "test copy")) if not first.uncertainties: - result = induce_program(workflows, propose=propose) if propose else first + result = ( + induce_program(workflows, propose=propose, **spoken) if propose else first + ) result.probe_target = name return result @@ -481,6 +538,9 @@ def probe_and_induce( if identities: scorer = _Scorer(bundle) points = optional_points(workflows) + narrated = { + ng.location: ng for ng in first.narrated_guards if ng.status == "proposed" + } for u in first.uncertainties: point = points.get(u.location) if point is None or point.kind != u.kind: @@ -503,9 +563,15 @@ def probe_and_induce( point=point, work_dir=Path(work_dir), runs=runs, + narrated=narrated.get(point.location), ) verdicts.append(verdict) - if verdict.certified: + if verdict.certified == NARRATED: + # Pin the exact guard that was probed; a later pass that + # proposes a different one is not certified by this probe. + ng = narrated[point.location] + resolutions[point.location] = f"{NARRATED}@{ng.guard_id}" + elif verdict.certified: resolutions[point.location] = verdict.certified try: target.reset() # leave the copy at its baseline @@ -513,7 +579,7 @@ def probe_and_induce( pass result = induce_program( - workflows, propose=propose, probe_resolutions=resolutions or None + workflows, propose=propose, probe_resolutions=resolutions or None, **spoken ) result.probe_target = name result.probe_runs = runs diff --git a/packages/flow/openadapt_flow/ir.py b/packages/flow/openadapt_flow/ir.py index dca06b1fd..3859e965b 100644 --- a/packages/flow/openadapt_flow/ir.py +++ b/packages/flow/openadapt_flow/ir.py @@ -901,6 +901,27 @@ class Guard(BaseModel): Predicate.model_rebuild() # resolve the self-referential `operands` +class GuardSource(BaseModel): + """Why a guard is in the program, when spoken narration suggested it. + + Written by induction (``compiler.narration_rules``) for a guard that + started as a narrated proposal and was then certified. The sentence is + scrubbed narration. It explains the guard to a reviewer; the runtime never + reads it. + """ + + sentence: str = Field(description="The scrubbed sentence that suggested it") + rule: str = Field(default="", description="The guard in plain words") + certified_by: Literal["recordings_agree", "probe", "person"] + accepted_by: Optional[str] = Field( + default=None, description="Who accepted it, when a person certified it" + ) + recording: Optional[int] = Field( + default=None, description="Index of the recording the sentence came from" + ) + source: Literal["narration", "narration+model"] = "narration" + + def predicate_contract_sha256(predicate: Optional[Predicate]) -> str: """Return the canonical contract digest for one program transition guard.""" @@ -2720,6 +2741,11 @@ class Workflow(BaseModel): # review, canary or admission pins. None for every other bundle, and # omitted from its serialized bytes. origin: Optional[Literal["agent_run"]] = None + # Narration that suggested a certified guard, keyed by the program state + # (or step) id the guard sits on. Explanation only: the runtime never reads + # it. Empty for every bundle without narrated guards, and then omitted + # from its serialized bytes. + guard_sources: dict[str, GuardSource] = Field(default_factory=dict) # Declared compensating actions, offered to a person after a stop and # never run automatically (``openadapt_flow.after_stop``). Part of the # content digest when present; omitted from the serialized bytes when @@ -2769,6 +2795,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: data.pop("surfaces", None) if self.origin is None: data.pop("origin", None) + if not self.guard_sources: + data.pop("guard_sources", None) if not self.undo_steps: data.pop("undo_steps", None) return data diff --git a/packages/flow/openadapt_flow/learning/branch_teach.py b/packages/flow/openadapt_flow/learning/branch_teach.py index 00c09a974..7301b8abe 100644 --- a/packages/flow/openadapt_flow/learning/branch_teach.py +++ b/packages/flow/openadapt_flow/learning/branch_teach.py @@ -33,6 +33,13 @@ lifecycle as any repair: campaigns, human approval and a canary (``openadapt-flow repair``). +Narration (``narration.json`` beside a fix recording or a halted run, see +:mod:`openadapt_flow.narration`) can name the narrowing parameter: "if the +payer type is medicaid, dismiss the notice". It is used only when every example +has that value and every normal run that shows the signature has another one. +Otherwise it is reported as contradicted and teach-branch decides as before. +The sentence is kept beside the guard (``Workflow.guard_sources``). + At runtime the branch is an ordinary guarded program state: an OCR text check and, when narrowed, a parameter compare. It makes zero model calls. """ @@ -43,12 +50,13 @@ import json import re from pathlib import Path -from typing import Callable, Optional, Sequence +from typing import Any, Callable, Optional, Sequence from pydantic import BaseModel, Field from openadapt_flow.ir import ( Guard, + GuardSource, ParamKind, Predicate, PredicateKind, @@ -156,6 +164,12 @@ class BranchTeachResult(BaseModel): negatives: int = 0 outcome: Optional[LearnOutcome] = None out_bundle: Optional[Path] = None + narration: Optional[str] = Field( + default=None, description="The narrated sentence that named the condition" + ) + narration_status: Optional[str] = Field( + default=None, description="'certified' (every run agrees) | 'contradicted'" + ) def summary(self) -> str: lines = [ @@ -165,6 +179,13 @@ def summary(self) -> str: ] if self.signature is not None: lines.append(f" condition: {self.signature.describe()}") + if self.narration is not None: + verdict = ( + "every example and normal run agrees" + if self.narration_status == "certified" + else "a run contradicts it, so it was not used" + ) + lines.append(f" narration: {self.narration!r} ({verdict})") if self.reason: lines.append(f" reason: {self.reason}") if self.outcome is not None and self.outcome.gate is not None: @@ -408,6 +429,71 @@ def _narrowing_param( return usable[0], "" +def _example_narration(example: BranchExample, extra: Sequence) -> list[str]: + """Sentences spoken while demonstrating this example's fix (or its run).""" + from openadapt_flow.narration import NarrationRefused, load_narration + + texts: list[str] = [] + for where in (Path(example.fix), Path(example.run_dir)): + if not where.is_dir(): + continue + try: + nar = load_narration(where) + except (NarrationRefused, ValueError): + nar = None + if nar is not None: + texts.extend(line.text for line in nar.lines) + for nar in extra: + texts.extend(line.text for line in nar.lines) + return texts + + +def _narrated_param( + sentences: Sequence[str], + examples: Sequence["_Example"], + base: Workflow, + others: Sequence[dict[str, str]] = (), +) -> Optional[tuple[str, str, str]]: + """``(param, value, sentence)`` a narrated rule names, or None. + + Only a categorical parameter (see :func:`_categorical`) whose value some + example or normal run (``others``) had can be named. Whether the runs + agree is checked by the caller. + """ + from openadapt_flow.compiler.narration_rules import ( + FieldValues, + _match_field, + parse_rule, + ) + + runs = [ex.params for ex in examples] + list(others) + names = sorted({k for r in runs for k in r if _categorical(base, k)}) + fields = [ + FieldValues(param=k, label=k, values=[r.get(k) for r in runs]) for k in names + ] + for sentence in sentences: + rule = parse_rule(sentence) + if rule is None or not rule.runs_when_condition_holds: + continue + matched = _match_field(rule.subject, rule.value, fields) + if matched is not None: + return matched[0].param, matched[1], rule.sentence + return None + + +def _narrated_guard_state(workflow: Workflow, predicate: Predicate) -> Optional[str]: + graph = workflow.program + if graph is None: + return None + for sid, state in graph.states.items(): + if any(tr.guard == predicate for tr in state.transitions): + return sid + if state.step is not None and state.step.guard is not None: + if state.step.guard.predicate == predicate: + return sid + return None + + def _happy_path(program: ProgramGraph) -> list[str]: """Intents along the unconditional first edges from the entry.""" intents: list[str] = [] @@ -442,13 +528,16 @@ def teach_branch( skill_id: Optional[str] = None, library_dir: Optional[Path | str] = None, screen_texts: Optional[Callable[[Path, RunReport], list[str]]] = None, + narration: Sequence = (), ) -> BranchTeachResult: """Learn one guarded branch from two or more resolved halts. Returns a :class:`BranchTeachResult`. ``promoted`` False is a governed refusal: nothing is written to ``out``. Raises :class:`TeachError` for unusable inputs. ``screen_texts`` reads the text of a normal run's screens - (default: OCR of its recorded frames). + (default: OCR of its recorded frames). ``narration`` adds + :class:`~openadapt_flow.narration.Narration` objects to the + ``narration.json`` files found beside the fixes and halted runs. """ if len(examples) < 2: raise TeachError( @@ -508,18 +597,39 @@ def teach_branch( clean = [ c for c in candidates if not any(_contains(t, c) for _, t in negative_runs) ] - if clean: - signature = BranchSignature(text=clean[0]) + text = clean[0] if clean else candidates[0] + overlapping = [ + runtime_params_for_gui(r.params) for r, t in negative_runs if _contains(t, text) + ] + sentences = [line for ex in examples for line in _example_narration(ex, ())] + [ + line.text for nar in narration for line in nar.lines + ] + spoken = _narrated_param( + sentences, + loaded, + base, + [runtime_params_for_gui(r.params) for r, _ in negative_runs], + ) + spoken_ok = ( + spoken is not None + and all(ex.params.get(spoken[0]) == spoken[1] for ex in loaded) + and all(n.get(spoken[0]) not in (None, spoken[1]) for n in overlapping) + ) + said: dict[str, Any] = { + "narration": spoken[2] if spoken else None, + "narration_status": ( + None if spoken is None else "certified" if spoken_ok else "contradicted" + ), + } + if spoken_ok: + assert spoken is not None + signature = BranchSignature(text=text, param=spoken[0], value=spoken[1]) + elif clean: + signature = BranchSignature(text=text) else: - text = candidates[0] - overlapping = [ - runtime_params_for_gui(r.params) - for r, t in negative_runs - if _contains(t, text) - ] narrowed, why = _narrowing_param(loaded, overlapping, base) if narrowed is None: - return _refused(why, examples=len(loaded), negatives=n_neg) + return _refused(why, examples=len(loaded), negatives=n_neg, **said) signature = BranchSignature(text=text, param=narrowed[0], value=narrowed[1]) # One correction trace per example, all keyed on the shared signature. @@ -626,6 +736,22 @@ def teach_branch( "data_sources": dict(base.data_sources), } ) + if spoken_ok: + assert spoken is not None + where = _narrated_guard_state(updated, signature.predicate()) + if where is not None: + updated = updated.model_copy( + update={ + "guard_sources": { + **updated.guard_sources, + where: GuardSource( + sentence=spoken[2], + rule=f"take the fix only when {signature.describe()}", + certified_by="recordings_agree", + ), + } + } + ) out_bundle = Path(out) _copy_templates(Path(bundle), out_bundle) _copy_carried(carried, out_bundle) @@ -638,6 +764,7 @@ def teach_branch( "condition": { "text_sha256": _digest(signature.text), "param": signature.param, + "narration_sha256": _digest(spoken[2]) if spoken_ok and spoken else None, }, "example_run_reports_sha256": [ _digest((ex.run_dir / "report.json").read_text(encoding="utf-8")) @@ -663,4 +790,5 @@ def teach_branch( negatives=n_neg, outcome=outcome, out_bundle=out_bundle, + **said, ) diff --git a/packages/flow/openadapt_flow/narration.py b/packages/flow/openadapt_flow/narration.py new file mode 100644 index 000000000..9936c73d9 --- /dev/null +++ b/packages/flow/openadapt_flow/narration.py @@ -0,0 +1,265 @@ +"""Spoken narration lined up with a recording's steps. + +A recording directory (``meta.json`` + ``events.jsonl`` + ``frames/``) may +carry ``narration.json``: what the person said while demonstrating, cut into +sentences, scrubbed of patient data, and lined up with the recorded steps. +The compiler uses it to **propose** a branch rule (see +:mod:`openadapt_flow.compiler.narration_rules`); a proposal never certifies +itself. + +The file is the same for every backend. It only needs the ``t`` (seconds since +the recording started) that every ``events.jsonl`` line carries, so web, +native, RDP and Citrix recordings get the same treatment. It arrives in one of +two ways: + +* ``openadapt-flow`` converts an openadapt-capture session that recorded with + ``--audio`` (:mod:`openadapt_flow.adapters.capture`); capture has already + scrubbed the text. +* ``openadapt-flow narration attach RECORDING --audio FILE.wav`` (or + ``--transcript FILE.json``) transcribes on this machine and scrubs here. + +Rules: + +* **Scrubbed or not written.** :func:`attach_narration` refuses without an + ``openadapt-privacy`` scrubber. :func:`load_narration` refuses a file that + does not say it was scrubbed. +* **Local only.** Transcription uses openadapt-capture's on-device backends. +* **Alignment is by time.** A sentence belongs to the next action the person + takes (people say what they are about to do), or to the last action when + they speak after the final one. +""" + +from __future__ import annotations + +import json +import os +import tempfile +from pathlib import Path +from typing import Any, Iterable, Optional, Sequence + +from pydantic import BaseModel, Field + +NARRATION_FILE = "narration.json" +NARRATION_SCHEMA = "openadapt.flow.narration/v1" + +#: A sentence that starts this soon after an action began still belongs to it +#: (the person kept talking as they clicked). +LEAD_S = 0.3 + +#: Event kinds that are steps (structural events such as switch_surface are not). +_NON_STEP_KINDS = frozenset({"switch_surface"}) + + +class NarrationRefused(RuntimeError): + """Narration was not written or read (not scrubbed, or no scrubber).""" + + +class NarrationLine(BaseModel): + """One spoken sentence, timed from the start of the recording.""" + + t_start: float = Field(description="Seconds since the recording started") + t_end: float + text: str = Field(description="Scrubbed sentence") + step_id: Optional[str] = Field( + default=None, + description="The step it is lined up with (step_)", + ) + + +class Narration(BaseModel): + """The ``narration.json`` sidecar.""" + + schema_id: str = Field(default=NARRATION_SCHEMA, alias="schema") + scrubbed: bool = True + scrubbed_by: dict[str, Any] = Field(default_factory=dict) + source: str = Field(default="", description="openadapt-capture | attach") + lines: list[NarrationLine] = Field(default_factory=list) + + model_config = {"populate_by_name": True} + + def for_step(self, step_id: str) -> list[NarrationLine]: + return [line for line in self.lines if line.step_id == step_id] + + +def step_id_for_event(index: int) -> str: + """The compiler's step id for ``events.jsonl`` event ``index``.""" + return f"step_{index:03d}" + + +def read_events(recording_dir: Path) -> list[dict[str, Any]]: + path = Path(recording_dir) / "events.jsonl" + if not path.is_file(): + raise FileNotFoundError(f"no events.jsonl in {recording_dir}") + return [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + + +def align( + sentences: Iterable[tuple[float, float, str]], + events: Sequence[dict[str, Any]], +) -> list[NarrationLine]: + """Line each ``(t_start, t_end, text)`` up with a recorded step. + + Times are seconds since the recording started (the events' ``t``). A + sentence goes to the first step whose ``t`` is at or after the sentence's + start minus :data:`LEAD_S`; with none, to the last step. + """ + steps = sorted( + ( + (float(e["t"]), int(e["i"])) + for e in events + if "t" in e and "i" in e and e.get("kind") not in _NON_STEP_KINDS + ), + key=lambda pair: pair[0], + ) + out: list[NarrationLine] = [] + for start, end, text in sentences: + text = str(text or "").strip() + if not text: + continue + chosen: Optional[int] = None + for t, i in steps: + if t >= start - LEAD_S: + chosen = i + break + if chosen is None and steps: + chosen = steps[-1][1] + out.append( + NarrationLine( + t_start=round(float(start), 3), + t_end=round(float(end), 3), + text=text, + step_id=step_id_for_event(chosen) if chosen is not None else None, + ) + ) + return out + + +def write_narration(directory: Path, narration: Narration) -> Path: + """Write ``narration.json`` atomically. Refuses unscrubbed narration.""" + if narration.scrubbed is not True: + raise NarrationRefused("narration is written only after it is scrubbed") + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + path = directory / NARRATION_FILE + fd, tmp = tempfile.mkstemp(dir=directory, prefix=".narration-", suffix=".json") + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(narration.model_dump_json(indent=2, by_alias=True) + "\n") + os.replace(tmp, path) + except BaseException: + Path(tmp).unlink(missing_ok=True) + raise + return path + + +def load_narration(path: Path | str) -> Optional[Narration]: + """Read ``narration.json`` from a recording or bundle directory, or None.""" + file = Path(path) / NARRATION_FILE + if not file.is_file(): + return None + data = json.loads(file.read_text(encoding="utf-8")) + if data.get("schema") != NARRATION_SCHEMA: + raise NarrationRefused(f"{file} is not a narration file") + if data.get("scrubbed") is not True: + raise NarrationRefused(f"{file} does not say it was scrubbed; not reading it") + return Narration.model_validate(data) + + +def _scrubber_evidence(scrubber: Any) -> dict[str, Any]: + from importlib import metadata + + try: + version = metadata.version("openadapt-privacy") + except metadata.PackageNotFoundError: + version = None + return { + "class": f"{type(scrubber).__module__}.{type(scrubber).__qualname__}", + "openadapt_privacy_version": version, + } + + +def require_scrubber() -> Any: + """The openadapt-privacy scrubber, or :class:`NarrationRefused`.""" + from openadapt_flow import privacy + + scrubber = privacy.get_scrubber() + if scrubber is None: + raise NarrationRefused( + "Narration is not stored without scrubbing: spoken words can name a " + "patient. Install openadapt-privacy with pip install " + "'openadapt-flow[privacy]' and python -m spacy download en_core_web_sm." + ) + return scrubber + + +def attach_narration( + recording_dir: Path | str, + sentences: Iterable[tuple[float, float, str]], + *, + scrubber: Any = None, + source: str = "attach", +) -> Path: + """Scrub ``sentences``, line them up with the recording, and write them. + + ``sentences`` are ``(t_start, t_end, text)`` in seconds since the + recording started. ``scrubber`` defaults to openadapt-privacy; without it + this raises :class:`NarrationRefused` and writes nothing. + """ + recording_dir = Path(recording_dir) + scrubber = scrubber if scrubber is not None else require_scrubber() + events = read_events(recording_dir) + clean = [] + for start, end, text in sentences: + try: + scrubbed = scrubber.scrub_text(str(text)) + except Exception as exc: # noqa: BLE001 - not scrubbed means not written + raise NarrationRefused( + f"the scrubber failed, nothing written: {exc}" + ) from exc + clean.append((start, end, scrubbed)) + narration = Narration( + scrubbed=True, + scrubbed_by=_scrubber_evidence(scrubber), + source=source, + lines=align(clean, events), + ) + return write_narration(recording_dir, narration) + + +def sentences_from_transcript( + result: dict[str, Any], *, offset: float = 0.0 +) -> list[tuple[float, float, str]]: + """Whisper-style result -> ``(t_start, t_end, text)`` sentences. + + ``offset`` is where the audio starts, in seconds since the recording + started (0 when audio and recording started together). + """ + try: + from openadapt_capture.narration import utterances_from_transcript + except ImportError: # capture extra missing: one sentence per segment + return [ + ( + float(s.get("start", 0.0)) + offset, + float(s.get("end", 0.0)) + offset, + str(s.get("text") or "").strip(), + ) + for s in result.get("segments") or [] + if str(s.get("text") or "").strip() + ] + return [ + (u.start, u.end, u.text) + for u in utterances_from_transcript(result, offset=offset) + ] + + +def transcribe_audio( + path: Path | str, *, model: str = "base", backend: str = "auto" +) -> dict[str, Any]: + """Transcribe an audio file on this machine (openadapt-capture's backends).""" + from openadapt_capture.narration import transcribe_file + + return transcribe_file(path, backend=backend, model=model) diff --git a/packages/flow/openadapt_flow/repair/cli.py b/packages/flow/openadapt_flow/repair/cli.py index 27e9d8dd2..cff7af507 100644 --- a/packages/flow/openadapt_flow/repair/cli.py +++ b/packages/flow/openadapt_flow/repair/cli.py @@ -130,12 +130,34 @@ def cmd_show(args: argparse.Namespace) -> int: except RepairLifecycleError as exc: return _print_refusal(exc) print(candidate.diff_summary()) + _print_narrated_guards(candidate) if args.json: print() print(candidate.model_dump_json(indent=2)) return 0 +def _print_narrated_guards(candidate: RepairCandidate) -> None: + """Show each guard the proposed bundle took from narration, with the + sentence (read from the local bundle; the candidate record holds digests).""" + path = candidate.proposed_bundle_path + if not path or not (Path(path) / "workflow.json").is_file(): + return + try: + workflow = Workflow.load(Path(path)) + except Exception: # noqa: BLE001 - the summary above is still valid + return + if not workflow.guard_sources: + return + print(" guards suggested by narration:") + for where, source in sorted(workflow.guard_sources.items()): + who = f" by {source.accepted_by}" if source.accepted_by else "" + print(f" - {where}: {source.rule or 'guard'}") + print( + f" said: {source.sentence!r} (certified: {source.certified_by}{who})" + ) + + def cmd_review(args: argparse.Namespace) -> int: store = _store(args) try: diff --git a/packages/flow/openadapt_flow/visualize/builder.py b/packages/flow/openadapt_flow/visualize/builder.py index 69480983b..01a93637b 100644 --- a/packages/flow/openadapt_flow/visualize/builder.py +++ b/packages/flow/openadapt_flow/visualize/builder.py @@ -518,6 +518,11 @@ def _add_edge( ] ) node.halts.append("halts until an authenticated answer is signed") + source = workflow.guard_sources.get(sid) if not prefix else None + if source is not None: + node.guard_reason = source.sentence + node.guard_certified_by = source.certified_by + node.badges.append("narrated rule") nodes.append(node) # exception handler edge if state.on_exception: @@ -557,6 +562,8 @@ def _add_edge( tr.label or (guard_summary or ""), guard=guard_summary, ) + if source is not None and tr.guard is not None: + edges[-1].guard_reason = source.sentence def _ordered_state_ids(graph: "ProgramGraph") -> list[str]: diff --git a/packages/flow/openadapt_flow/visualize/projection.py b/packages/flow/openadapt_flow/visualize/projection.py index 83b3b1f90..647c78d51 100644 --- a/packages/flow/openadapt_flow/visualize/projection.py +++ b/packages/flow/openadapt_flow/visualize/projection.py @@ -89,6 +89,7 @@ class ProjectionBoundaryError(RuntimeError): _RISKS: Final = frozenset({"reversible", "irreversible"}) _OUTCOMES: Final = frozenset({"success", "halt", "escalate"}) _GUARD_ON_UNMET: Final = frozenset({"halt", "skip"}) +_GUARD_CERTIFIED_BY: Final = frozenset({"recordings_agree", "probe", "person"}) _POSTCONDITION_KINDS: Final = frozenset( { "text_present", @@ -124,6 +125,7 @@ class ProjectionBoundaryError(RuntimeError): "API", "secret", "optional (skippable)", + "narrated rule", "loop", "human decision", "halt", @@ -175,6 +177,7 @@ class ProjectionBoundaryError(RuntimeError): "has_api_binding", # bool "postconditions", # closed vocabulary "guard_on_unmet", # closed vocabulary + "guard_certified_by", # closed vocabulary "outcome", # closed vocabulary "halts", # REPLACED by _safe_halts (positional, no content) "badges", # closed literal set + bounded count template @@ -188,6 +191,7 @@ class ProjectionBoundaryError(RuntimeError): "key", # recorded key "api_summary", # method + URL template "guard", # free-text predicate summary + "guard_reason", # spoken sentence (scrubbed, still recorded text) "wait_until", # free-text predicate summary "reason", # free-text terminal reason } @@ -245,7 +249,8 @@ class ProjectionBoundaryError(RuntimeError): _BUNDLE_LOCAL: Final = frozenset({"created_at", "viewport"}) _EDGE_PUBLIC: Final = frozenset({"source", "target", "kind", "label"}) -_EDGE_LOCAL: Final = frozenset({"guard"}) # free-text predicate summary +# guard: free-text predicate summary; guard_reason: spoken sentence. +_EDGE_LOCAL: Final = frozenset({"guard", "guard_reason"}) _SPEC_PUBLIC: Final = frozenset({"spec_version", "bundle", "nodes", "edges"}) _SPEC_LOCAL: Final[frozenset[str]] = frozenset() @@ -467,6 +472,7 @@ def _project_node(node: GraphNode) -> GraphNode: kind for kind in node.postconditions if kind in _POSTCONDITION_KINDS ], guard_on_unmet=_in_vocabulary(node.guard_on_unmet, _GUARD_ON_UNMET), + guard_certified_by=_in_vocabulary(node.guard_certified_by, _GUARD_CERTIFIED_BY), outcome=_in_vocabulary(node.outcome, _OUTCOMES), halts=_safe_halts(node), badges=_safe_badges(node.badges), diff --git a/packages/flow/openadapt_flow/visualize/render.py b/packages/flow/openadapt_flow/visualize/render.py index bed680ed8..28f8bd2a0 100644 --- a/packages/flow/openadapt_flow/visualize/render.py +++ b/packages/flow/openadapt_flow/visualize/render.py @@ -102,6 +102,7 @@ def render_mermaid(spec: ProgramGraphSpec) -> str: irrev: list[str] = [] halt: list[str] = [] id_map: dict[str, str] = {} + reasons: list[str] = [] for i, node in enumerate(spec.nodes): nid = f"n{i}" id_map[node.id] = nid @@ -141,6 +142,8 @@ def render_mermaid(spec: ProgramGraphSpec) -> str: if node.risk == "irreversible": badges.append("irreversible") irrev.append(nid) + if node.guard_reason: + badges.append("narrated rule") suffix = ( f"
{_mm(' · '.join(badges), 40)}" if badges else "" ) @@ -155,9 +158,12 @@ def render_mermaid(spec: ProgramGraphSpec) -> str: if node.outcome in ("halt", "escalate"): halt.append(nid) else: - lines.append(f' {nid}{{"{label}"}}') + extra = "
narrated rule" if node.guard_reason else "" + lines.append(f' {nid}{{"{label}{extra}"}}') if node.halts: halt.append(nid) + if node.guard_reason: + reasons.append(f" %% {nid} because: {_mm(node.guard_reason, 160)}") for edge in spec.edges: src = id_map.get(edge.source) tgt = id_map.get(edge.target) @@ -168,6 +174,7 @@ def render_mermaid(spec: ProgramGraphSpec) -> str: lines.append(f" {src} -->|{label}| {tgt}") else: lines.append(f" {src} --> {tgt}") + lines.extend(reasons) lines.append(" classDef irreversible stroke:#b4530a,stroke-width:2px;") lines.append(" classDef halt stroke:#b21f2d,stroke-width:2px;") if irrev: diff --git a/packages/flow/openadapt_flow/visualize/spec.py b/packages/flow/openadapt_flow/visualize/spec.py index a78100816..d488641e4 100644 --- a/packages/flow/openadapt_flow/visualize/spec.py +++ b/packages/flow/openadapt_flow/visualize/spec.py @@ -251,6 +251,14 @@ class GraphNode(BaseModel): guard_on_unmet: Optional[str] = Field( default=None, description="halt|skip when guard unmet" ) + guard_reason: Optional[str] = Field( + default=None, + description="The scrubbed spoken sentence that suggested this guard", + ) + guard_certified_by: Optional[str] = Field( + default=None, + description="recordings_agree|probe|person: what certified a narrated guard", + ) wait_until: Optional[str] = Field( default=None, description="Readiness predicate summary" ) @@ -282,6 +290,10 @@ class GraphEdge(BaseModel): guard: Optional[str] = Field( default=None, description="Guard summary for a branch edge" ) + guard_reason: Optional[str] = Field( + default=None, + description="The scrubbed spoken sentence that suggested this branch", + ) class ProgramGraphSpec(BaseModel): diff --git a/packages/flow/openadapt_flow/visualize/static/program_graph.js b/packages/flow/openadapt_flow/visualize/static/program_graph.js index 977e397d7..21914daf9 100644 --- a/packages/flow/openadapt_flow/visualize/static/program_graph.js +++ b/packages/flow/openadapt_flow/visualize/static/program_graph.js @@ -182,6 +182,7 @@ "guard", node.guard + " → " + (node.guard_on_unmet || "halt") ); + narratedRow(detail, node); if (node.param) detailRow(detail, "input", "parameter " + node.param); if (detail.childNodes.length) card.appendChild(detail); @@ -211,6 +212,25 @@ return card; } + var CERTIFIED_BY = { + recordings_agree: "every recording agrees", + probe: "confirmed on a test copy", + person: "accepted by a person", + }; + + function narratedRow(detail, node) { + if (!node.guard_reason) return; + detailRow( + detail, + "because", + "\u201c" + + node.guard_reason + + "\u201d (said while recording; certified: " + + (CERTIFIED_BY[node.guard_certified_by] || "unknown") + + ")" + ); + } + function renderControlNode(node) { var card = el("div", "opg-node"); var head = el("div", "opg-node-head"); @@ -231,6 +251,11 @@ }); card.appendChild(chips); } + if (node.guard_reason) { + var why = el("div", "opg-detail"); + narratedRow(why, node); + card.appendChild(why); + } return card; } diff --git a/packages/flow/schemas/program-graph-v2.json b/packages/flow/schemas/program-graph-v2.json index 2293a76b8..71f1aca99 100644 --- a/packages/flow/schemas/program-graph-v2.json +++ b/packages/flow/schemas/program-graph-v2.json @@ -224,6 +224,19 @@ "default": null, "description": "Guard summary for a branch edge", "title": "Guard" + }, + "guard_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The scrubbed spoken sentence that suggested this branch", + "title": "Guard Reason" } }, "required": [ @@ -409,6 +422,32 @@ "description": "halt|skip when guard unmet", "title": "Guard On Unmet" }, + "guard_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The scrubbed spoken sentence that suggested this guard", + "title": "Guard Reason" + }, + "guard_certified_by": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "recordings_agree|probe|person: what certified a narrated guard", + "title": "Guard Certified By" + }, "wait_until": { "anyOf": [ { diff --git a/packages/flow/tests/test_narration_rules.py b/packages/flow/tests/test_narration_rules.py new file mode 100644 index 000000000..a7222c06d --- /dev/null +++ b/packages/flow/tests/test_narration_rules.py @@ -0,0 +1,951 @@ +"""Spoken narration proposes a branch guard; recordings, a probe or a person +certify it. + +Synthetic data only: invented payers and amounts, a fake scrubber, and (in one +optional test) synthetic speech made with macOS ``say``. +""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path + +import pytest + +from openadapt_flow.compiler.induction import induce_program +from openadapt_flow.compiler.narration_rules import parse_rule +from openadapt_flow.ir import ActionKind, PredicateKind, StateKind, Step, Workflow +from openadapt_flow.narration import ( + Narration, + NarrationLine, + NarrationRefused, + align, + attach_narration, + load_narration, +) +from openadapt_flow.runtime.replayer import Replayer +from tests.test_replayer import FakeBackend, FakeVision + +RULE = "If the insurance is Medicare, skip the copay screen." + + +def _type(sid: str, field: str, value: str, *, risk: str = "reversible") -> Step: + return Step( + id=sid, intent=f"type {field}", action=ActionKind.TYPE, text=value, risk=risk + ) + + +def _key(sid: str, key: str, intent: str, *, risk: str = "reversible") -> Step: + return Step(id=sid, intent=intent, action=ActionKind.KEY, key=key, risk=risk) + + +def visit(payer: str, copay: bool) -> Workflow: + """Check in a visit. The copay is a consequential write, done for some payers.""" + steps = [ + _type("step_000", "patient", "Test Patient"), + _type("step_001", "insurance", payer), + ] + if copay: + steps.append(_type("step_002", "copay", "20", risk="irreversible")) + steps.append(_key("step_003", "Enter", "press Enter to finish")) + return Workflow(name="check-in", steps=steps) + + +def said(text: str, step_id: str) -> Narration: + return Narration( + lines=[NarrationLine(t_start=1.0, t_end=3.0, text=text, step_id=step_id)] + ) + + +def _opt(result): + return next( + s + for s in result.program.states.values() + if s.kind is StateKind.ACTION and s.step and s.step.intent == "type copay" + ) + + +def _run(workflow, tmp_path, params): + backend = FakeBackend() + bundle = tmp_path / "bundle" + bundle.mkdir(exist_ok=True) + report = Replayer(backend, vision=FakeVision(), poll_interval_s=0.01).run( + workflow, + bundle_dir=bundle, + run_dir=tmp_path / f"run-{len(list(tmp_path.iterdir()))}", + params=params, + ) + return backend, report + + +# -- parsing ------------------------------------------------------------------ + + +@pytest.mark.parametrize( + "sentence,value,skip,runs", + [ + (RULE, "Medicare", True, False), + ( + "Skip the copay screen when the insurance is Medicare.", + "Medicare", + True, + False, + ), + ("Since it's Medicare, I don't fill in the copay.", "Medicare", True, False), + ( + "If the insurance is not Medicare, enter the copay.", + "Medicare", + False, + False, + ), + ("Unless the payer is Aetna, leave the referral blank.", "Aetna", True, True), + ( + "If it says Prior Auth Required, click Request Auth.", + "Prior Auth Required", + False, + True, + ), + ], +) +def test_sentences_parse_into_condition_and_action(sentence, value, skip, runs): + rule = parse_rule(sentence) + assert rule is not None + assert (rule.value, rule.skip, rule.runs_when_condition_holds) == ( + value, + skip, + runs, + ) + + +def test_plain_narration_is_not_a_rule(): + assert parse_rule("Now I click save.") is None + assert parse_rule("Open the chart for the patient.") is None + + +# -- required cases ----------------------------------------------------------- + + +def test_a_narrated_branch_becomes_a_proposed_guard(): + """Insurance is on screen but never typed: the sentence maps to a screen + element. Nothing checked it, so it is a flagged proposal and the point + stays refused.""" + traces = [ + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _key("step_003", "Enter", "finish"), + ], + ), + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _type("step_002", "copay", "20", risk="irreversible"), + _key("step_003", "Enter", "finish"), + ], + ), + ] + result = induce_program(traces, narration=[said(RULE, "step_003"), None]) + (ng,) = result.narrated_guards + assert ng.status == "proposed" and ng.certified_by is None + assert ng.guard.kind is PredicateKind.NOT + assert ng.guard.operands[0].kind is PredicateKind.TEXT_PRESENT + assert ng.guard.operands[0].text == "Medicare" + assert ng.sentence == RULE + assert not result.certified and result.program is None + (u,) = result.uncertainties + assert u.proposal is not None and u.proposal.trusted is False + assert u.proposal.source == "narration" + assert any(o.key == "narrated" for o in u.question.options) + assert "--accept-narrated opt_" in u.record_next + + +def test_the_guard_certifies_when_every_recording_agrees(tmp_path): + traces = [visit("Medicare", False), visit("Aetna", True), visit("Cigna", True)] + result = induce_program(traces, narration=[said(RULE, "step_003"), None, None]) + (ng,) = result.narrated_guards + assert ng.status == "certified" and ng.certified_by == "recordings_agree" + assert ng.agree == [0, 1, 2] + assert result.certified, result.render() + state = _opt(result) + guard = state.step.guard + assert guard.on_unmet == "skip" + assert guard.predicate.kind is PredicateKind.NOT + inner = guard.predicate.operands[0] + assert (inner.kind, inner.value) == (PredicateKind.PARAM_EQUALS, "Medicare") + source = result.workflow.guard_sources[state.id] + assert source.sentence == RULE and source.certified_by == "recordings_agree" + + # The compiled program follows the rule on new runs. + pname = inner.param + params = {k: v for k, v in result.workflow.params.items()} + backend, report = _run(result.workflow, tmp_path, {**params, pname: "Medicare"}) + assert report.success and ("type", "20") not in backend.actions + backend, report = _run(result.workflow, tmp_path, {**params, pname: "Humana"}) + assert report.success and ("type", "20") in backend.actions + + +def test_a_contradicting_recording_keeps_it_refused(): + traces = [ + visit("Medicare", False), + visit("Aetna", True), + visit("Medicare", True), # did the copay for Medicare after all + ] + first = induce_program(traces, narration=[said(RULE, "step_003"), None, None]) + result = induce_program( + traces, + narration=[said(RULE, "step_003"), None, None], + accepted_narration={f"opt_2@{first.narrated_guards[0].guard_id}": "reviewer"}, + ) + (ng,) = result.narrated_guards + assert ng.status == "contradicted" and ng.contradict == [2] + assert not result.certified and result.program is None + (u,) = result.uncertainties + assert u.kind == "unconfirmed_optional" + assert "not used" in u.detail + assert not any(o.key == "narrated" for o in u.question.options) + + +def test_healthy_runs_make_no_model_calls(tmp_path): + class CountingInterpreter: + source = "narration+model" + calls = 0 + + def interpret(self, sentence, context): + CountingInterpreter.calls += 1 + return None + + traces = [visit("Medicare", False), visit("Aetna", True)] + result = induce_program( + traces, + narration=[said(RULE, "step_003"), None], + interpreter=CountingInterpreter(), + ) + assert result.certified + # The parser read the sentence, so the compile-time model was not asked. + assert CountingInterpreter.calls == 0 + pname = result.narrated_guards[0].guard.operands[0].param + for payer in ("Medicare", "Aetna"): + _, report = _run( + result.workflow, tmp_path, {**result.workflow.params, pname: payer} + ) + assert report.success + assert report.model_calls == 0 + assert report.est_model_cost_usd == 0.0 + + +def test_the_transcript_is_scrubbed_before_it_is_stored(tmp_path): + rec = tmp_path / "rec" + rec.mkdir() + (rec / "events.jsonl").write_text( + "\n".join( + json.dumps(e) + for e in [ + {"i": 0, "kind": "type", "t": 1.0}, + {"i": 1, "kind": "key", "t": 4.0}, + ] + ) + + "\n" + ) + + class Scrubber: + def scrub_text(self, text, is_separated=False): + return text.replace("John Smith", "") + + sentences = [(0.2, 0.9, "Patient John Smith."), (2.0, 3.5, RULE)] + from openadapt_flow import privacy + + privacy.set_text_scrubber(None) + try: + with pytest.raises(NarrationRefused): + attach_narration(rec, sentences) + finally: + privacy.reset_scrubbers() + assert not (rec / "narration.json").exists() + + attach_narration(rec, sentences, scrubber=Scrubber()) + raw = (rec / "narration.json").read_text() + assert "John Smith" not in raw + nar = load_narration(rec) + assert [(line.text, line.step_id) for line in nar.lines] == [ + ("Patient .", "step_000"), + (RULE, "step_001"), + ] + data = json.loads(raw) + data["scrubbed"] = False + (rec / "narration.json").write_text(json.dumps(data)) + with pytest.raises(NarrationRefused): + load_narration(rec) + + +# -- the other ways to certify -------------------------------------------------- + + +def test_a_person_can_accept_a_guard_nothing_contradicts(): + traces = [ + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _key("step_003", "Enter", "finish"), + ], + ), + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _type("step_002", "copay", "20", risk="irreversible"), + _key("step_003", "Enter", "finish"), + ], + ), + ] + first = induce_program(traces, narration=[said(RULE, "step_003"), None]) + pinned = f"opt_1@{first.narrated_guards[0].guard_id}" + # A bare location does not accept anything: the id pins the exact guard. + bare = induce_program( + traces, + narration=[said(RULE, "step_003"), None], + accepted_narration={"opt_1": "R. Reviewer"}, + ) + assert not bare.certified + result = induce_program( + traces, + narration=[said(RULE, "step_003"), None], + accepted_narration={pinned: "R. Reviewer"}, + ) + (ng,) = result.narrated_guards + assert (ng.status, ng.certified_by, ng.accepted_by) == ( + "certified", + "person", + "R. Reviewer", + ) + assert result.certified + assert result.workflow.guard_sources["opt_1"].accepted_by == "R. Reviewer" + + +def test_a_screen_element_guard_certifies_from_the_recorded_screens(): + traces = [ + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _key("step_003", "Enter", "finish"), + ], + ), + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _type("step_002", "copay", "20", risk="irreversible"), + _key("step_003", "Enter", "finish"), + ], + ), + ] + screens = {0: ["Coverage: Medicare Part B"], 1: ["Coverage: Aetna PPO"]} + result = induce_program( + traces, + narration=[said(RULE, "step_003"), None], + screen_texts=lambda t, step_id: screens[t], + ) + (ng,) = result.narrated_guards + assert ng.status == "certified" and ng.certified_by == "recordings_agree" + assert result.certified + + +def test_a_narrated_divergence_becomes_a_branch(tmp_path): + def trace(payer: str) -> Workflow: + decide = ( + _key("step_002", "M", "press M to bill medicare", risk="irreversible") + if payer == "Medicare" + else _key( + "step_002", "C", "press C to collect the copay", risk="irreversible" + ) + ) + return Workflow( + name="bill", + steps=[ + _type("step_001", "insurance", payer), + decide, + _key("step_003", "Enter", "finish"), + ], + ) + + plain = induce_program([trace("Medicare"), trace("Aetna")]) + assert not plain.certified # a divergence with no detectable condition + + rule = "If the insurance is Medicare, bill Medicare directly." + result = induce_program( + [trace("Medicare"), trace("Aetna"), trace("Cigna")], + narration=[said(rule, "step_002"), None, None], + ) + (ng,) = result.narrated_guards + assert ng.location.startswith("divergence_") and ng.status == "certified" + assert result.certified, result.render() + branch = next( + s for s in result.program.states.values() if s.kind is StateKind.BRANCH + ) + assert branch.id in result.workflow.guard_sources + pname = ng.guard.param + backend, report = _run( + result.workflow, tmp_path, {**result.workflow.params, pname: "Medicare"} + ) + assert report.success and ("press", "M") in backend.actions + backend, report = _run( + result.workflow, tmp_path, {**result.workflow.params, pname: "Humana"} + ) + assert report.success and ("press", "C") in backend.actions + + +def test_a_sentence_about_another_step_is_not_used(): + traces = [visit("Medicare", False), visit("Aetna", True)] + result = induce_program( + traces, + narration=[ + said("If the insurance is Medicare, skip the referral.", "step_003"), + None, + ], + ) + assert result.narrated_guards == [] + assert not result.certified + + +# -- every backend -------------------------------------------------------------- + + +@pytest.mark.parametrize("backend", ["web", "native", "rdp", "citrix"]) +def test_narration_is_the_same_for_every_backend(tmp_path, backend): + """Narration only uses the events' ``t`` clock, so a web, native, RDP or + Citrix recording of the same demonstration lines up and proposes the same + guard.""" + + class Scrubber: + def scrub_text(self, text, is_separated=False): + return text + + def recording(name: str, wf: Workflow, sentences) -> Path: + rec = tmp_path / backend / name + rec.mkdir(parents=True) + (rec / "meta.json").write_text(json.dumps({"id": name, "backend": backend})) + events = [ + {"i": int(s.id.split("_")[1]), "kind": s.action.value, "t": 2.0 * k} + for k, s in enumerate(wf.steps) + ] + (rec / "events.jsonl").write_text( + "\n".join(json.dumps(e) for e in events) + "\n" + ) + if sentences: + attach_narration(rec, sentences, scrubber=Scrubber()) + return rec + + traces = [visit("Medicare", False), visit("Aetna", True)] + recs = [ + recording("medicare", traces[0], [(3.2, 4.0, RULE)]), + recording("aetna", traces[1], []), + ] + narration = [load_narration(r) for r in recs] + assert narration[0].lines[0].step_id == "step_003" + result = induce_program(traces, narration=narration) + (ng,) = result.narrated_guards + assert (ng.status, ng.certified_by, ng.location) == ( + "certified", + "recordings_agree", + "opt_2", + ) + + +def test_alignment_attaches_a_sentence_to_the_next_action(): + events = [{"i": 0, "t": 1.0}, {"i": 1, "t": 5.0}, {"i": 2, "t": 9.0}] + lines = align([(0.0, 0.5, "a."), (4.9, 6.0, "b."), (12.0, 13.0, "c.")], events) + assert [line.step_id for line in lines] == ["step_000", "step_001", "step_002"] + + +def test_capture_narration_is_carried_into_the_recording(tmp_path): + from types import SimpleNamespace + + from openadapt_flow.adapters.capture import _write_capture_narration + + session = SimpleNamespace( + narration=lambda: [SimpleNamespace(start=103.0, end=104.0, text=RULE)] + ) + _write_capture_narration( + session, + tmp_path, + 100.0, + [{"i": 0, "t": 1.0, "kind": "type"}, {"i": 1, "t": 3.5, "kind": "key"}], + ) + nar = load_narration(tmp_path) + assert nar.source == "openadapt-capture" + assert [(line.t_start, line.step_id) for line in nar.lines] == [(3.0, "step_001")] + + +# -- real local speech (optional) --------------------------------------------- + +FIXTURE = ( + Path(__file__).resolve().parents[2] + / "capture" + / "tests" + / "fixtures" + / "narration" + / "medicare_copay.wav" +) + + +@pytest.mark.skipif( + importlib.util.find_spec("faster_whisper") is None or not FIXTURE.is_file(), + reason="faster-whisper or the synthetic speech fixture is not available", +) +def test_synthetic_speech_end_to_end(tmp_path): + """Synthetic speech -> local Whisper -> scrub -> align -> induce. Skipped + unless the tiny.en model is already cached (never downloads).""" + try: + from huggingface_hub import try_to_load_from_cache + + cached = try_to_load_from_cache("Systran/faster-whisper-tiny.en", "model.bin") + except Exception: # noqa: BLE001 + cached = None + if not isinstance(cached, str): + pytest.skip("tiny.en Whisper model is not cached locally") + from openadapt_flow.narration import sentences_from_transcript, transcribe_audio + + class Scrubber: + def scrub_text(self, text, is_separated=False): + return text.replace("John Smith", "") + + traces = [visit("Medicare", False), visit("Aetna", True)] + rec = tmp_path / "rec" + rec.mkdir() + events = [ + {"i": int(s.id.split("_")[1]), "kind": "x", "t": 3.0 * k} + for k, s in enumerate(traces[0].steps) + ] + (rec / "events.jsonl").write_text("\n".join(json.dumps(e) for e in events) + "\n") + result = transcribe_audio( + FIXTURE, model=str(Path(cached).parent), backend="faster-whisper" + ) + attach_narration( + rec, sentences_from_transcript(result, offset=1.5), scrubber=Scrubber() + ) + assert "john smith" not in (rec / "narration.json").read_text().lower() + induced = induce_program(traces, narration=[load_narration(rec), None]) + (ng,) = induced.narrated_guards + assert ng.status == "certified" and induced.certified + + +# -- probing on a test copy ----------------------------------------------------- + + +def test_probing_confirms_a_narrated_guard_the_recordings_cannot_check( + tmp_path, monkeypatch +): + """Coverage shows on screen and is never typed, so the recordings cannot + check "skip the copay when it's Medicare". The scripted test copy can: + the narrated reading produces the recorded effect in both contexts and + dropping the copay does not.""" + from types import SimpleNamespace + + from openadapt_flow.compiler import probing + from openadapt_flow.reward.models import RewardBundle + + coverage = {"p-med": "Medicare", "p-aet": "Aetna"} + + class ScreenVision(FakeVision): + def __init__(self, patient): + super().__init__() + self.patient = patient + + def text_present(self, screen_png, text, **_kw): + return text.lower() in f"coverage: {coverage[self.patient]}".lower() + + class TestCopy: + name = "scripted-test-copy" + + def __init__(self): + self.last: dict[str, bool] = {} + + def reset(self): + pass + + def run(self, workflow, *, bundle_dir, run_dir, params): + patient = params["patient_id"] + backend = FakeBackend() + report = Replayer( + backend, vision=ScreenVision(patient), poll_interval_s=0.01 + ).run(workflow, bundle_dir=bundle_dir, run_dir=run_dir, params=params) + copay = ("type", "20") in backend.actions + # The recorded effect: a copay exactly when coverage is not Medicare. + self.last[patient] = report.success and copay == ( + coverage[patient] != "Medicare" + ) + return probing.RunSignal( + probing.runtime_signal_of(report), report.model_calls + ) + + target = TestCopy() + + class Scorer: + def __init__(self, bundle): + pass + + def baseline(self, identity): + return None + + def judge(self, identity, signal, before): + ok = target.last.get(identity["patient_id"], False) + return SimpleNamespace( + outcome=SimpleNamespace(value="verified" if ok else "wrong_effect"), + reason="", + ) + + bundle = SimpleNamespace( + identity_keys=("patient_id",), check_identity=lambda i: None + ) + monkeypatch.setattr(probing, "_Scorer", Scorer) + monkeypatch.setattr(RewardBundle, "load", classmethod(lambda cls, x: x)) + + def recording(patient, copay): + steps = [_type("step_000", "patient", patient)] + if copay: + steps.append(_type("step_002", "copay", "20", risk="irreversible")) + steps.append(_key("step_003", "Enter", "finish")) + return probing.ProbeTrace( + workflow=Workflow(name="c", steps=steps, params={"patient_id": patient}), + bundle_dir=tmp_path / patient, + label=patient, + ) + + traces = [recording("p-med", False), recording("p-aet", True)] + plain = induce_program( + [t.workflow for t in traces], narration=[said(RULE, "step_003"), None] + ) + assert plain.narrated_guards[0].status == "proposed" and not plain.certified + + result = probing.probe_and_induce( + traces, + target=target, + reward_contract=bundle, + work_dir=tmp_path / "probe", + narration=[said(RULE, "step_003"), None], + ) + (verdict,) = result.probe_verdicts + assert verdict.certified == "narrated", verdict + assert verdict.interpretations["narrated"] == "passed" + assert verdict.interpretations["incidental"] == "failed" + (ng,) = result.narrated_guards + assert (ng.status, ng.certified_by) == ("certified", "probe") + assert result.certified + assert all(run.model_calls == 0 for run in result.probe_runs) + + +# -- CLI ------------------------------------------------------------------------ + + +def _screen_only_traces(): + return [ + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _key("step_003", "Enter", "finish"), + ], + ), + Workflow( + name="c", + steps=[ + _type("step_000", "patient", "A"), + _type("step_002", "copay", "20", risk="irreversible"), + _key("step_003", "Enter", "finish"), + ], + ), + ] + + +def test_cli_induce_accepts_a_narrated_guard_only_with_a_name(tmp_path, capsys): + from openadapt_flow.__main__ import main + from openadapt_flow.narration import write_narration + + dirs = [] + for i, wf in enumerate(_screen_only_traces()): + d = tmp_path / f"b{i}" + wf.save(d) + dirs.append(str(d)) + write_narration(Path(dirs[0]), said(RULE, "step_003")) + out = tmp_path / "out" + + assert main(["induce", *dirs, "--out", str(out)]) == 2 + report = capsys.readouterr().out + assert "[narrated]" in report and not out.exists() + import re + + pinned = re.search(r"--accept-narrated (opt_1@[0-9a-f]+)", report).group(1) + + assert main(["induce", *dirs, "--out", str(out), "--accept-narrated", pinned]) == 2 + assert "--accepted-by" in capsys.readouterr().out + rc = main( + [ + "induce", + *dirs, + "--out", + str(out), + "--accept-narrated", + "opt_1", + "--accepted-by", + "R. Reviewer", + ] + ) + assert rc == 2 and "LOCATION@GUARD_ID" in capsys.readouterr().out + + rc = main( + [ + "induce", + *dirs, + "--out", + str(out), + "--accept-narrated", + pinned, + "--accepted-by", + "R. Reviewer", + ] + ) + assert rc == 0, capsys.readouterr().out + written = Workflow.load(out) + (source,) = written.guard_sources.values() + assert (source.sentence, source.accepted_by) == (RULE, "R. Reviewer") + + +def test_cli_narration_attach_scrubs_a_transcript(tmp_path, capsys): + from openadapt_flow import privacy + from openadapt_flow.__main__ import main + + rec = tmp_path / "rec" + rec.mkdir() + (rec / "events.jsonl").write_text( + json.dumps({"i": 0, "kind": "type", "t": 2.0}) + "\n" + ) + transcript = tmp_path / "t.json" + transcript.write_text( + json.dumps( + { + "segments": [ + { + "start": 0.5, + "end": 1.5, + "text": "Patient John Smith is Medicare.", + } + ] + } + ) + ) + + class Scrubber: + def scrub_text(self, text, is_separated=False): + return text.replace("John Smith", "") + + privacy.set_text_scrubber(None) + try: + assert ( + main(["narration", "attach", str(rec), "--transcript", str(transcript)]) + == 2 + ) + assert not (rec / "narration.json").exists() + privacy.set_text_scrubber(Scrubber()) + assert ( + main(["narration", "attach", str(rec), "--transcript", str(transcript)]) + == 0 + ) + finally: + privacy.reset_scrubbers() + out = capsys.readouterr().out + assert "John Smith" not in out and "" in out + assert "John Smith" not in (rec / "narration.json").read_text() + + +# -- explain the program back ------------------------------------------------- + + +def test_visualize_shows_each_guard_with_the_sentence_that_suggested_it(tmp_path): + from openadapt_flow.visualize import ( + PresentationProfile, + build_program_graph, + project_program_graph, + render_html, + render_mermaid, + ) + + traces = [visit("Medicare", False), visit("Aetna", True)] + result = induce_program(traces, narration=[said(RULE, "step_003"), None]) + result.workflow.save(tmp_path / "prog") + spec = build_program_graph(Workflow.load(tmp_path / "prog")) + (node,) = [n for n in spec.nodes if n.guard_reason] + assert node.guard_reason == RULE + assert node.guard_certified_by == "recordings_agree" + assert "narrated rule" in node.badges + assert "insurance" in node.guard and node.guard_on_unmet == "skip" + assert RULE in render_html(spec) + assert "narrated rule" in render_mermaid(spec) and "because:" in render_mermaid( + spec + ) + + # Off this machine the sentence is recorded text, so it is dropped. + for profile in ("remote-safe", "public-synthetic"): + safe = project_program_graph(spec, PresentationProfile(profile)) + assert all(n.guard_reason is None for n in safe.nodes) + assert all(e.guard_reason is None for e in safe.edges) + assert any(n.guard_certified_by == "recordings_agree" for n in safe.nodes) + + +def test_visualize_shows_a_narrated_branch_on_its_edge(): + from openadapt_flow.visualize import build_program_graph + + def trace(payer: str) -> Workflow: + decide = ( + _key("step_002", "M", "press M to bill medicare", risk="irreversible") + if payer == "Medicare" + else _key( + "step_002", "C", "press C to collect the copay", risk="irreversible" + ) + ) + return Workflow(name="b", steps=[_type("step_001", "insurance", payer), decide]) + + rule = "If the insurance is Medicare, bill Medicare directly." + result = induce_program( + [trace("Medicare"), trace("Aetna")], narration=[said(rule, "step_002"), None] + ) + spec = build_program_graph(result.workflow) + (branch,) = [n for n in spec.nodes if n.kind.value == "branch"] + assert branch.guard_reason == rule + assert [e.guard_reason for e in spec.edges if e.guard_reason] == [rule] + + +def test_a_refused_capture_narration_is_left_out_not_fatal(tmp_path): + from types import SimpleNamespace + + from openadapt_flow.adapters.capture import _write_capture_narration + + def refuse(): + raise ValueError("narration.json does not say it was scrubbed") + + _write_capture_narration(SimpleNamespace(narration=refuse), tmp_path, 0.0, []) + assert load_narration(tmp_path) is None + + +# -- regressions from review ---------------------------------------------------- + + +def test_recorded_values_are_compared_exactly_like_the_runtime(): + """ "medicare" and "Medicare" are different run-time values, so a recording + that typed one must not vouch for the other.""" + traces = [visit("Medicare", False), visit("medicare", False), visit("Aetna", True)] + result = induce_program(traces, narration=[said(RULE, "step_003"), None, None]) + (ng,) = result.narrated_guards + assert ng.status == "contradicted" and ng.contradict == [1] + assert not result.certified + + +def test_a_divergence_with_a_step_inside_it_stays_refused(): + def trace(steps): + return Workflow( + name="d", steps=[_type("step_000", "insurance", steps[0])] + steps[1] + ) + + a = _key("step_001", "B", "press B for beta", risk="irreversible") + z = _key("step_003", "Z", "press Z for zeta", risk="irreversible") + al = _key("step_001", "A", "press A for alpha", risk="irreversible") + om = _key("step_003", "O", "press O for omega", risk="irreversible") + note = _type("step_002", "note", "n") + traces = [ + trace(("Medicare", [a, z])), + trace(("Aetna", [al, om])), + trace(("Medicare", [a, note, z])), + ] + rule = "If the insurance is Medicare, press beta and zeta." + result = induce_program(traces, narration=[said(rule, "step_001"), None, None]) + assert not result.certified + + +@pytest.mark.parametrize( + "sentence,runs", + [ + ("Skip the copay if the insurance is Medicare, otherwise enter it.", False), + ("If it's not Medicare, skip the copay.", True), + ("If it's Medicare, the copay isn't needed.", False), + ], +) +def test_negations_parse_the_right_way_round(sentence, runs): + rule = parse_rule(sentence) + assert rule is not None and rule.value == "Medicare" + assert rule.runs_when_condition_holds is runs + + +def test_an_unclear_double_negative_is_not_a_rule(): + assert parse_rule("Don't skip the copay if it's Medicare.") is None + + +def test_model_answers_are_validated(): + from openadapt_flow.compiler.narration_rules import ( + FieldValues, + candidate_from_model, + ) + + fields = [ + FieldValues(param="insurance", label="insurance", values=["Medicare", "Aetna"]) + ] + kw = dict( + sentence="Medicare ones go straight through", + trace=0, + step_id=None, + fields=fields, + source="m", + ) + # A string "false" is not a boolean; the answer is dropped. + assert ( + candidate_from_model( + {"field": "insurance", "value": "Medicare", "step_runs_when_true": "false"}, + **kw, + ) + is None + ) + # A field that was not offered, or a value no recording had, is dropped. + assert ( + candidate_from_model( + {"field": "payer", "value": "Medicare", "step_runs_when_true": False}, **kw + ) + is None + ) + assert ( + candidate_from_model( + {"field": "insurance", "value": "Cigna", "step_runs_when_true": False}, **kw + ) + is None + ) + # Screen text must be words the person said. + assert ( + candidate_from_model( + {"text": "Prior auth", "value": "x", "step_runs_when_true": True}, **kw + ) + is None + ) + ok = candidate_from_model( + {"field": "insurance", "value": "medicare", "step_runs_when_true": False}, **kw + ) + assert ( + ok is not None + and ok.condition.value == "Medicare" + and not ok.runs_when_condition_holds + ) + + +def test_a_malformed_narration_file_is_skipped_by_the_cli(tmp_path, capsys): + from openadapt_flow.__main__ import main + + dirs = [] + for i, wf in enumerate([visit("Medicare", False), visit("Aetna", True)]): + d = tmp_path / f"b{i}" + wf.save(d) + dirs.append(str(d)) + (Path(dirs[0]) / "narration.json").write_text( + '{"schema": "openadapt.flow.narration/v1", "scrubbed": true, "lines": [{"t_start": "x"}]}' + ) + assert main(["induce", *dirs, "--out", str(tmp_path / "out")]) == 2 + assert "Narration not used" in capsys.readouterr().out diff --git a/packages/flow/tests/test_teach_branch_narration.py b/packages/flow/tests/test_teach_branch_narration.py new file mode 100644 index 000000000..dd08dd9c4 --- /dev/null +++ b/packages/flow/tests/test_teach_branch_narration.py @@ -0,0 +1,157 @@ +"""teach-branch: narration names the parameter that narrows the branch. + +Same MockMed prior-auth scenario as ``test_teach_branch``. Two enum +parameters separate the examples from the normal run, so without narration +teach-branch refuses. A narrated sentence picks one, and is used only when +every example and normal run agrees with it. Synthetic data only. +""" + +from __future__ import annotations + +from openadapt_flow.ir import ParamKind, ParamSpec, Workflow +from openadapt_flow.learning.branch_teach import BranchExample, teach_branch +from openadapt_flow.narration import Narration, NarrationLine, write_narration +from tests.test_teach_branch import ( + AETNA, + CIGNA, + HUMANA, + _bundle, + _dismissed, + _fix, + _halt, + _normal_texts, + _replay, + _workflow, +) + +RULE = "If the payer type is medicaid, dismiss the prior auth notice." + + +def _two_param_workflow() -> Workflow: + wf = _workflow() + return wf.model_copy( + update={ + "param_specs": { + "payer_type": ParamSpec( + name="payer_type", + type=ParamKind.ENUM, + choices=["medicaid", "commercial"], + example="commercial", + ), + "plan_tier": ParamSpec( + name="plan_tier", + type=ParamKind.ENUM, + choices=["silver", "gold"], + example="gold", + ), + } + } + ) + + +def _setup(tmp_path, sentence): + wf = _two_param_workflow() + bundle = _bundle(tmp_path, wf) + fix = _fix(tmp_path) + branch = {"payer_type": "medicaid", "plan_tier": "silver"} + normal = {"payer_type": "commercial", "plan_tier": "gold"} + r1 = _halt(wf, bundle, tmp_path / "r1", AETNA, branch) + r2 = _halt(wf, bundle, tmp_path / "r2", CIGNA, branch) + if sentence: + write_narration( + r1, Narration(lines=[NarrationLine(t_start=0.0, t_end=2.0, text=sentence)]) + ) + normal_dir = tmp_path / "normal" + report, _ = _replay( + wf, bundle, normal_dir, text=AETNA, blocking=False, params=normal + ) + assert report.success + return ( + wf, + bundle, + [BranchExample(run_dir=r1, fix=fix), BranchExample(run_dir=r2, fix=fix)], + normal_dir, + ) + + +def test_without_narration_two_separating_parameters_are_refused(tmp_path): + _, bundle, examples, normal_dir = _setup(tmp_path, None) + result = teach_branch( + examples, + tmp_path / "out", + bundle=bundle, + negatives=[normal_dir], + screen_texts=_normal_texts({normal_dir: [AETNA]}), + ) + assert not result.promoted + assert "several parameters" in result.reason + + +def test_narration_picks_the_parameter_when_every_run_agrees(tmp_path): + _, bundle, examples, normal_dir = _setup(tmp_path, RULE) + out = tmp_path / "out" + result = teach_branch( + examples, + out, + bundle=bundle, + negatives=[normal_dir], + screen_texts=_normal_texts({normal_dir: [AETNA]}), + ) + assert result.promoted, result.summary() + assert (result.signature.param, result.signature.value) == ( + "payer_type", + "medicaid", + ) + assert (result.narration, result.narration_status) == (RULE, "certified") + assert "narration:" in result.summary() + learned = Workflow.load(out) + (source,) = learned.guard_sources.values() + assert source.sentence == RULE and source.certified_by == "recordings_agree" + report, backend = _replay( + learned, + out, + tmp_path / "m", + text=HUMANA, + params={"payer_type": "medicaid", "plan_tier": "gold"}, + ) + assert report.success and _dismissed(backend) + assert report.model_calls == 0 + + +def test_narration_a_run_contradicts_is_not_used(tmp_path): + # The examples were silver; the sentence says gold. + _, bundle, examples, normal_dir = _setup( + tmp_path, "If the plan tier is gold, dismiss the notice." + ) + result = teach_branch( + examples, + tmp_path / "out", + bundle=bundle, + negatives=[normal_dir], + screen_texts=_normal_texts({normal_dir: [AETNA]}), + ) + assert not result.promoted + assert result.narration_status == "contradicted" + assert "several parameters" in result.reason + + +def test_repair_review_shows_the_sentence_beside_the_guard(tmp_path, capsys): + from openadapt_flow.repair.cli import _print_narrated_guards + from openadapt_flow.repair.registration import load_detached_candidate + + _, bundle, examples, normal_dir = _setup(tmp_path, RULE) + out = tmp_path / "out" + assert teach_branch( + examples, + out, + bundle=bundle, + negatives=[normal_dir], + screen_texts=_normal_texts({normal_dir: [AETNA]}), + ).promoted + candidate = load_detached_candidate(out) + _print_narrated_guards(candidate) + shown = capsys.readouterr().out + assert "guards suggested by narration" in shown + assert RULE in shown and "recordings_agree" in shown + # The candidate record itself still carries digests, not the sentence. + assert RULE not in candidate.model_dump_json()