flow: run the API step in durable runs without ever resending a save

CAPABILITIES.md

@@ -182,6 +182,7 @@ improvising or retrying when something is off.
 | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` |
 | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) |
 | Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) |
+| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) |
 | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |
 | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |
 | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |

llms-full.txt

@@ -538,6 +538,7 @@ improvising or retrying when something is off.
 | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` |
 | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) |
 | Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) |
+| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) |
 | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) |
 | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) |
 | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) |

packages/flow/docs/API_TIER.md

@@ -133,16 +133,50 @@ An application that issues a new session value on every form read can't be
 approved this way: the second read changes the digest, and the item stops
 before saving.
 
+## In a durable run
+
+A durable run (`--durable`) uses the API step too. The checkpoints and the
+resume rules keep the no-double-write rule. (Standard and Regulated runs are
+durable and governed; the governed exclusion below still applies to them.)
+
+- **Before the span**, the checkpoint is the last step before the steps the
+  API step covers.
+- **After the span**, the covered steps (`api_folded`) and the save (`api`)
+  are checkpointed together, once the save was sent. If the save wasn't
+  sent (for example, the record check's pre-state couldn't be read, or an
+  approval stopped the item), nothing in the span is checkpointed. A resume
+  then runs the whole API step again from the start: it prepares the
+  request again, asks again under an approval mode, and sends it once.
+- **After a send that may have landed** (a lost response, a rejected
+  response, or a save the record check didn't confirm), the pause is at the
+  save and the run reads `RECONCILIATION_REQUIRED` when the outcome is
+  unknown. Resume never sends that save again and never redoes it through
+  the GUI on an empty form. An ordinary resume is refused, as for any
+  uncertain delivery. A resume that authorizes one uncertain retry stops at
+  the save with that reason. The attended `reconcile` action reads the record
+  back with the record check and finishes the run when the record is there,
+  without sending anything.
+- A run that fell back to the GUI steps and stopped at the GUI save resumes
+  that GUI save, as before.
+
+Reconcile reads the record as it is now. A record check that compares with
+the state before the save (`exact_new_set`, `count_new_only` or a
+collateral-loss guard) can't be proven that way after the fact, so the pause
+stays for a person to check the record. The same limit applies to a GUI
+save.
+
 ## When it doesn't run
 
 The API step runs only in a plain linear run. It doesn't run, and the GUI
 steps run instead, when:
 
 - an approval mode is on and the API actuator can't prepare the request
   before sending it (a deployment's own actuator without `prepare`);
-- the run is governed by qualification, is durable, or is resumed;
+- the run is governed by qualification;
 - the step isn't admitted, or the program's steps, on-screen checks or the
   save's record check changed since it was proposed or admitted;
+- the save checks the right record on screen and the API step names no
+  right-record contract;
 - there's no live browser session (`--backend web` only).
 
 `Replayer(api_steps=False)` keeps a run on the GUI.

packages/flow/openadapt_flow/action_evidence.py

@@ -19,6 +19,9 @@
     {"uia", "dom", "guarded_coordinate", "guarded_keyboard", "remote_guarded"}
 )
 HUMAN_ATTENDED_ACTUATIONS = frozenset({"human_attended", "human_attended_skip"})
+#: A GUI step that an admitted API step ran in place of: nothing was
+#: delivered for it; the API step's save carries the delivery evidence.
+API_FOLDED_ACTUATION = "api_folded"
 
 
 _DELIVERY_OPERATIONS: dict[tuple[str, ActionKind], set[tuple[str, bool]]] = {
@@ -308,7 +311,8 @@ def action_evidence_error(
             or result.input_verified is not None
             or result.starting_state_settled not in {None, True}
             or result.fresh_actuation_events
-            or result.actuation not in {None, "human_attended_skip"}
+            or result.actuation
+            not in {None, "human_attended_skip", API_FOLDED_ACTUATION}
         ):
             return "skipped action contains delivery evidence"
         return None

packages/flow/openadapt_flow/runtime/replayer.py

@@ -2155,6 +2155,25 @@ def acknowledge_delivery(self_nonlocal) -> None:
                     continue
                 if step_index <= span_skip_to:
                     continue
+                if resume_from is not None and step_index == resume_from:
+                    inside = self._resume_inside_web_form_span(
+                        workflow, step_index, report.results
+                    )
+                    if inside is not None:
+                        # The folded steps are checkpointed only after the
+                        # save was sent, so this save may already have
+                        # landed. It is never sent again and never redone
+                        # through the GUI on an empty form.
+                        self._record_linear_result(
+                            report,
+                            workflow,
+                            inside,
+                            step_index=step_index,
+                            step=step,
+                            params=params,
+                            durable_run=durable_run,
+                        )
+                        break
                 # An admitted API step (docs/API_TIER.md) runs in place of the
                 # GUI steps it stands for. When it can't, the reason lands on
                 # the first of those steps and the GUI steps run as recorded.
@@ -2194,15 +2213,25 @@ def acknowledge_delivery(self_nonlocal) -> None:
                     if why_not is not None:
                         result.api_fallback = why_not
                     batch = [result]
-                for result in batch:
+                # An API span's folded steps are checkpointed only once its
+                # save was sent: a resume then starts at the save, which can
+                # only be reconciled, never redone through the GUI. A save
+                # that wasn't sent leaves the resume point before the span,
+                # so a resume runs the whole span again.
+                save_result = batch[-1]
+                checkpoint_folded = save_result.ok or save_result.delivery_attempted
+                for offset, result in enumerate(batch):
+                    folded = offset < len(batch) - 1
                     stop_run = self._record_linear_result(
                         report,
                         workflow,
                         result,
-                        step_index=step_index,
-                        step=step,
+                        step_index=step_index + offset,
+                        step=workflow.steps[step_index + offset],
                         params=params,
-                        durable_run=durable_run,
+                        durable_run=(
+                            None if folded and not checkpoint_folded else durable_run
+                        ),
                     )
                     if stop_run:
                         break
@@ -2403,8 +2432,6 @@ def _web_form_span_at(
             ids = [s.id for s in workflow.steps[step_index : end_index + 1]]
             if not self.api_steps:
                 reason = "API steps are off for this run"
-            elif durable:
-                reason = "a durable or resumed run uses the GUI steps"
             elif getattr(self, "_surface_router", None) is not None:
                 reason = "a run across several windows or tabs uses the GUI steps"
             elif self._approval_gated() and not callable(
@@ -2421,6 +2448,13 @@ def _web_form_span_at(
                 reason = "a governed run uses its qualified GUI path"
             elif not binding.admitted():
                 reason = "the API step is not admitted (openadapt-flow api admit)"
+            elif save.identity_armed and not binding.identity:
+                # The GUI save checks the right record on screen. The API step
+                # must check it too, so it needs a right-record contract.
+                reason = (
+                    "the save checks the right record on screen, and the API "
+                    "step names no right-record contract"
+                )
             elif list(binding.effects) != list(save.effects):
                 reason = (
                     "the save step's record check differs from the one admitted "
@@ -2444,6 +2478,54 @@ def _web_form_span_at(
             return end_index, save, reason
         return None
 
+    def _resume_inside_web_form_span(
+        self, workflow: Workflow, step_index: int, prior: list[StepResult]
+    ) -> Optional[StepResult]:
+        """A halt result when a resume would start inside an API step's span.
+
+        Returns None when ``step_index`` isn't a covered step after the first,
+        or when the steps before it ran on the GUI (``prior`` holds the
+        checkpointed results): then the GUI save resumes as usual.
+        """
+
+        step = workflow.steps[step_index]
+        if not prior or prior[-1].actuation != "api_folded":
+            return None
+        for save in workflow.steps[step_index:]:
+            binding = save.api_binding
+            if (
+                binding is None
+                or binding.kind != "web_form"
+                or binding.web_form is None
+            ):
+                continue
+            covered = list(binding.web_form.replaces_steps)
+            if step.id in covered[1:]:
+                return StepResult(
+                    step_id=step.id,
+                    intent=step.intent,
+                    ok=False,
+                    safety_halt=True,
+                    failure_category=(
+                        "governed_refusal"
+                        if self.governed_authorization is not None
+                        else "safety_halt"
+                    ),
+                    risk=step.risk,
+                    risk_explanation=step.risk_explanation,
+                    risk_review_required=step.risk_review_required,
+                    delivery_attempted=False,
+                    error=(
+                        f"Step '{step.id}' ({step.intent}): resume would start "
+                        f"inside the API step for '{save.id}', whose save was "
+                        "sent and may have landed. Resume never sends it again "
+                        "and never redoes it through the GUI. Reconcile it "
+                        "(the attended reconcile action reads the record back), "
+                        "or start the item again"
+                    ),
+                )
+        return None
+
     def _run_web_form_span(
         self,
         workflow: Workflow,