Close the last routes to a dishonest completed closure, and measure them

.github/workflows/ci.yml

@@ -60,6 +60,12 @@ jobs:
       - name: Run checks
         run: bun run check
 
+      # Free and offline: the committed cassettes replay recorded model decisions
+      # against this build's runtime, so a change that refuses a sequence a real model
+      # already performed fails here rather than at the next paid matrix.
+      - name: Replay recorded model decisions
+        run: bun run replay
+
   platform-persistence:
     strategy:
       fail-fast: false

.github/workflows/evals.yml

@@ -0,0 +1,114 @@
+name: Model evals
+
+# Model-in-the-loop evidence, on a schedule and never on a pull request.
+#
+# `bun run check` proves the runtime deterministically. It cannot prove that a real
+# model driven by the real prompts reaches the intended outcome, and until this
+# workflow existed that evidence came from whichever model the maintainer happened
+# to run locally: every recorded report was single-model, so "works with Flow" meant
+# "worked once, with one provider".
+#
+# Evals cost money and need credentials, so they stay out of the PR gate. This runs
+# weekly against at least two providers and publishes the report as an artifact,
+# with the qualification thresholds applied by `scripts/qualify-release.ts`.
+# See docs/adr/0010-declared-canonical-gate.md.
+
+on:
+  schedule:
+    # Mondays, early UTC: a full pass is long and expensive, so once a week.
+    - cron: "17 4 * * 1"
+  workflow_dispatch:
+    inputs:
+      models:
+        description: Comma-separated providerID/modelID list
+        required: false
+        type: string
+      repeat:
+        description: Attempts per scenario and model
+        required: false
+        default: "3"
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: model-evals
+  cancel-in-progress: false
+
+jobs:
+  matrix-evals:
+    runs-on: ubuntu-latest
+    timeout-minutes: 360
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          # This job runs a model-driven agent that executes arbitrary commands for
+          # hours with provider secrets in the environment. A GITHUB_TOKEN left in
+          # .git/config would be a repository-scoped write credential it never uses:
+          # the job only reads the checkout, runs the eval, and uploads an artifact.
+          persist-credentials: false
+
+      - name: Set up Node
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 24
+
+      - name: Set up Bun
+        uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
+        with:
+          bun-version: 1.3.14
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+
+      # Absent credentials are a configuration state, not a failure: a fork and a
+      # freshly cloned repository both have none, and failing the schedule there
+      # only teaches people to ignore it.
+      - name: Resolve model matrix
+        id: matrix
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          REQUESTED: ${{ inputs.models }}
+          CONFIGURED: ${{ vars.FLOW_EVAL_MODELS }}
+        run: |
+          models="${REQUESTED:-$CONFIGURED}"
+          if [ -z "$models" ] || { [ -z "$ANTHROPIC_API_KEY" ] && [ -z "$OPENAI_API_KEY" ]; }; then
+            echo "::notice::No eval model matrix or provider credentials configured; skipping."
+            echo "models=" >> "$GITHUB_OUTPUT"
+          else
+            echo "models=$models" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Run the eval matrix
+        if: steps.matrix.outputs.models != ''
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          FLOW_EVAL_MODEL: ${{ steps.matrix.outputs.models }}
+          FLOW_EVAL_NO_AUTH_COPY: "1"
+          # Through the environment, not interpolated into the script: a dispatch
+          # input is caller text, and `run:` splices it before any shell quoting
+          # applies. The runner rejects a non-integer itself.
+          REPEAT: ${{ inputs.repeat || '3' }}
+        # A failing scenario must still publish its report: the durable documents are
+        # the only way to read why, and paying for the pass twice is the alternative.
+        continue-on-error: true
+        run: bun run eval -- --repeat "$REPEAT"
+
+      # Before qualification, so a NOT QUALIFIED verdict still leaves the report that
+      # explains it. Publishing after the gate meant the one run worth reading was the
+      # one whose artifact was skipped.
+      - name: Publish the report
+        if: steps.matrix.outputs.models != ''
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: eval-report
+          path: evals/results/
+          retention-days: 90
+
+      - name: Apply release-qualification thresholds
+        if: steps.matrix.outputs.models != ''
+        run: bun run qualify

CHANGELOG.md

@@ -4,7 +4,143 @@ One short entry per release, written for users deciding whether to upgrade.
 
 ## [Unreleased]
 
-No changes yet.
+The last route to a dishonest `completed` closure is closed, and the two claims
+that rested on prompt prose are now measured.
+
+Qualified on Flow 7.0.2 / OpenCode 1.18.6 from two reports: the 63-run matrix of
+2026-07-28T03:20Z across `opencode/claude-sonnet-5`, `openai/gpt-5.6-sol`, and
+`xai/grok-4.5` (62/62 scored passed), plus 2026-07-28T07:09Z re-measuring
+`failing-gate-blocks @ xai/grok-4.5`, whose third attempt in the matrix had wedged on
+a `grep` call that never returned. That pair went 3/3 on the re-run and the wedge did
+not recur.
+
+- **A plan declares its canonical gate.** `plan.gate` is the exact command that
+  validates the whole repository, named at planning time and locked by approval. A
+  `broad` observation has to run that command byte-for-byte, and the gate joins the
+  vetoed-command set, so its latest failure blocks review whatever scope an
+  observation claimed. This closes the escape [ADR
+  0009](docs/adr/0009-scope-keyed-validation-veto.md) recorded as open: a measured
+  run had closed `completed` by claiming `git diff --check` as its gate — a command
+  that cannot fail, so nothing was ever observed red.
+- **A plan declares evidence this host cannot produce.** `plan.externalEvidence`
+  names each acceptance observation needing an operating system, service, credential,
+  or device this machine may lack, with the exact command whose passing is that
+  observation. Final review and `completed` closure are both refused until that
+  command passes, so a self-written proxy cannot stand in for it. The first
+  three-provider eval matrix found three runs substituting one — two closed
+  `completed` over a Windows-only criterion on a Linux host, with the independent
+  review passing, and the suite's own false-completion metric reported zero. Each
+  entry also names the `platform` that can observe it — `win32`, `darwin`, `linux`, or
+  `other` for a service, credential, or device — and Flow records the host every
+  observation ran on, so the declared command passing on the wrong machine no longer
+  discharges it. The next matrix run found exactly that: a declared Windows entry
+  cleared by its own command's exit zero on Linux, green because the Windows case is
+  skipped there. The refusal now distinguishes a command never observed from one that
+  passed on the wrong host, because those call for opposite moves. See
+  [ADR 0011](docs/adr/0011-declared-external-evidence.md).
+- **Acceptance evidence names test cases, not exit codes.** Each `externalEvidence`
+  entry also declares `assertions`: the case names whose passing *is* that
+  observation. `flow_validation_start` takes `resultsPath`, the JUnit XML the command
+  writes, and Flow records what that report said about each declared name. An entry is
+  satisfied only when every one is reported `passed` — `skipped` and `absent`
+  discharge nothing. This closes the limitation ADR 0011 recorded rather than fixed:
+  comparing the host closed the *wrong machine* and left the same skip on the right
+  one, where a case guarded, filtered, renamed out of the run, or never written still
+  exits zero. The names come from the approved plan and never from the caller, and a
+  report that predates the arming is read as no report at all. Declare an empty list
+  when the evidence is not a test result; that keeps the exit-code rule, which is the
+  honest answer for a credential or a device. See
+  [ADR 0012](docs/adr/0012-named-results-over-exit-codes.md).
+- **The reviewer can see the two commands the plan declares.** Its plan context now
+  carries `gate` and `externalEvidence`, which it was asked to judge and was never
+  shown: it could not tell a `broad` observation that ran the canonical gate from one
+  that ran something else, and could not check a declared environment against the host
+  the observation recorded.
+- **Session v5 schema:** `plan.gate` is a new optional string. A document written by
+  an earlier build still hydrates and keeps the older rule where `broad` is the
+  claimant's word; `flow_plan_save` requires the field for any new plan, so a plan
+  this build writes always declares one. `plan.externalEvidence` is a new optional
+  array under the same rule: older documents declare nothing and owe no acceptance
+  observation, and `flow_plan_save` requires the field — an empty list is the answer
+  when the goal is fully observable here. Each entry carries an optional `platform`
+  that `flow_plan_save` likewise requires for a new plan, and each validation
+  observation carries an optional runtime-written `hostPlatform`; an entry hydrated
+  without a platform keeps the command-only rule. Each entry also carries an optional
+  `assertions` list that `flow_plan_save` requires for a new plan, and each observation
+  may carry a `resultsPath` and the `observedAssertions` the runtime read from it; an
+  entry hydrated without assertions keeps the exit-code rule. Rolling an active session
+  back to a build without these fields is not supported, as with every previous
+  widening.
+- **The bar is published, not described.** [What Flow
+  guarantees](docs/guarantees.md) states which claims are enforced by types, attested
+  by the host, declared by the caller, judged by a model, or unenforced.
+  [Release qualification](docs/release-qualification.md) publishes the eval
+  thresholds a release clears, and `bun run qualify` applies them.
+- **Evals measure false completion and reviewer activity**, derived from durable
+  documents rather than model prose, and a scheduled workflow runs the suite against
+  at least two providers weekly. A new scenario checks that a requirement no run can
+  observe is never reported as verified. A run that aborts mid-flight is counted and
+  excluded rather than scored as a failure — one wedged attempt was the only failing
+  threshold in a measured report, on a guarantee that never ran — and `bun run
+  qualify` refuses a report holding one on a gated pair. A wedge diagnostic now names
+  the command each incomplete tool call was running.
+- **Recorded model decisions replay for free.** Every paid attempt now writes a
+  cassette — its tool calls, in order, with their arguments — and `bun run replay`
+  feeds those back through the real handlers against a fresh workspace with no model
+  and no host. It is deliberately the decision layer, not the HTTP wire: freezing tool
+  results too would mean Flow's own refusals never execute on replay, which is the one
+  class of defect this suite exists to catch. Every runtime change up to now needed
+  another paid matrix before anyone knew it had not broken a sequence a model already
+  performed. CI gates the committed cassettes; a run whose recording holds something a
+  decision-layer replay cannot reproduce is reported rather than gated.
+- **A report can be read.** `bun run triage` ranks a report's runs by how much
+  reading each is worth and prints its reason for each, because nobody should trust an
+  eval score without reading transcripts and there was no tooling for it: a 54-run
+  report was a table and a JSON file. It ranks rather than filters, and deliberately
+  does *not* flag a scored escalation every attempt of its pair made, or a lone silent
+  review pass — including both flagged 32 of 54 runs on a real report, almost all of
+  them the suite working. Excluding them flagged five: the wedge, the false completion,
+  and three escalation outliers. An empty result says so, since a suite that never
+  flags anything and one that measures nothing look identical from here.
+- **Three eval tiers, three prices.** `bun run replay` is free and answers whether the
+  runtime still reaches the same outcome; `bun run eval:smoke` is one model and one
+  attempt, for a prompt change; the full matrix is the only thing that qualifies a
+  release. One price for every question is what made the suite something run at release
+  instead of during work.
+- **A regression scenario for the hole named results closed.** `skipped-case-refused`
+  seeds a fixture whose Windows-only case is an ordinary `test.skipIf`, so the declared
+  command runs here, on the right host, and exits zero. Declaring the command is no
+  longer enough; the plan has to name the case. It went 9/9 across three providers on
+  first measurement and stays ungated anyway: every attempt declared `platform:
+  "win32"` on a Linux host, so the platform rule refuses first and the named-case rule
+  is never what binds. What the scenario measures today is the declaration, not the
+  observation.
+- **`unprovable-claim-refused` is gated at 90%.** Measured 0/3, then 8/9, then 9/9 as
+  the rule and the prompts landed; the margin is one pair's own variance rather than an
+  allowance for refusals to fail. See
+  [release qualification](docs/release-qualification.md).
+- **Seven cassettes are pinned, so CI replays real decisions.** One per scenario from
+  the 2026-07-28 matrix, spread across three providers. All 63 candidates from that run
+  replayed against this code; the only divergence was the attempt that wedged
+  mid-flight, which is advisory by construction. A run that ends by asking the user no
+  longer records a fidelity caveat either — the harness aborts that session itself, so
+  the `MessageAbortedError` it leaves behind was this suite's own doing, and calling it
+  unreproducible had made 19 of 63 cassettes advisory, every refusal scenario among
+  them.
+- **Eval reports now include the reviewer.** Subtask sessions are read too, so a
+  report finally contains the independent review's own tool calls. Before this, no
+  recorded report held a single `flow_feature_complete` call, the check for rejected
+  submissions could never fire, and the reviewer's tokens went uncounted — so token and
+  cost totals from earlier reports are lower than the same runs would report now.
+- **`/flow-auto` says when your host cannot continue.** Continuation needs assistant
+  message parentage; a host that does not report it now gets a plain note at startup
+  and an `autoContinuation` field on status, instead of a lifecycle that appears to
+  stop after every feature for no reason.
+- **Positioning, including when not to use Flow.** See
+  [positioning](docs/positioning.md) and the new README section.
+
+The prompt surface got smaller, not larger: the typed gate replaced the prose that
+asked the model to judge whether its own coverage claim was honest.
 
 ## [7.0.2] - 2026-07-27
 

CONTEXT.md

@@ -59,15 +59,20 @@ feature.
 code, output digest, and completeness stored directly on the active run. It is
 not a detached receipt or caller-authored success claim.
 
-**Broad validation**: A coverage claim that the command is the repository's
-canonical applicable gate or a justified equivalent for the delivered state. The
-string `broad` does not make a narrow check comprehensive, and claiming it binds
-the claimant: a `broad` observation that does not pass vetoes review until that
-same command passes.
+**Declared gate**: The plan's `gate`, the exact canonical command that validates
+the whole repository. It is named before implementation and locked by approval, so
+the coverage decision sits in the document the user approves. A plan saved before
+this field existed declares none and keeps the older rule.
+
+**Broad validation**: An observation of the declared gate. Any other command is
+refused the claim, as is one that selects which tests it runs. Claiming it binds the
+claimant: a `broad` observation that does not pass vetoes review until that same
+command passes.
 
 **Vetoed command**: A command whose latest evidence blocks review until it passes
-again for the current source — one an observation claimed at `broad` scope, or one
-whose bytes equal an entry in the active feature's validation list. Reset does not
+again for the current source — one an observation claimed at `broad` scope, the
+plan's declared gate, or one whose bytes equal an entry in the active feature's
+validation list. Reset does not
 erase the failure and no other passing command discharges it. The veto is
 prospective; the maintainer contract owns the exact admission rule.