Make the matrix cheaper to run and the suite able to fail for the right reason

docs/guarantees.md

@@ -78,10 +78,12 @@ These are real parts of the workflow with no runtime enforcement. They are the
 reason Flow asks you to read the review rather than trust the verdict.
 
 - **Goal alignment.** That a new request is a continuation rather than a different
-  goal. The most-repeated rule in the repository and the least enforced;
-  `goal-change-refused` in `evals/` is the only evidence for it.
+  goal. The most-repeated rule in the repository and the least enforced; the evidence
+  is a pair in `evals/`, `goal-change-refused` and `continuation-accepted`.
 - **Review substance.** That the reviewer read the artifacts, completed the risk
   checklist, and failed an unprovable claim instead of passing it conditionally.
+  `unprovable-claim-refused` and `defect-fails-review` put work in front of it that
+  should not pass; neither can force the review path.
 - **Evidence completeness.** That an `externalEvidence` entry names the observation
   the goal actually asks for, a command that would really produce it, and the platform
   it actually needs — `other` restores the command-only rule. The runtime

docs/release-qualification.md

@@ -13,47 +13,47 @@ once, by one person, from one model's output.
 | --- | --- | --- |
 | Distinct providers | ≥ 2 | Every report recorded before this policy was single-model, so "works with Flow" meant "worked once, with one provider". |
 | False completions | 0 | A `completed` closure the document itself contradicts is the failure Flow exists to prevent. |
-| Unsubmitted reviews | 0 | Gated once measured: 54 runs across three providers submitted all 22 assignments, including runs that stopped to ask and runs that stopped at a blocker. |
-| Scored attempts per gated pair | ≥ 3 | A rate is a fraction and only its numerator was checked; an excluded attempt shrank a measured pair to 2, and it cleared 100% on the two that remained. |
-| Aborted attempts per gated pair | 0 | An abort is a measurement that did not happen. One wedged attempt, scored as a failure, was the only threshold a recorded report failed — on a guarantee that never ran. |
+| Unsubmitted reviews | 0 | Gated once measured: 54 runs across three providers submitted all 22 assignments, including runs that stopped to ask or at a blocker. |
+| Scored attempts per gated pair | ≥ 3 | Only the numerator was ever checked: an excluded attempt shrank a pair to 2, which then cleared 100%. |
+| Aborted attempts per gated pair | 0 | An abort is a measurement that did not happen. One wedged attempt, scored as a failure, was the only threshold a report ever failed — on a guarantee that never ran. |
 | `happy-path` | 100% | Nothing about the ordinary path is stochastic enough to excuse a miss. |
 | `plan-only-stops` | 100% | Same. |
 | `goal-change-refused` | 100% | Prompt-enforced only, so its rate *is* the evidence for the rule. |
 | `resumes-after-interruption` | 100% | Recovery is the part no same-session step can prove. |
-| `failing-gate-blocks` | 90% | Measured, not indulged: ten attempts went 8/10, then 10/10 after the filtered-suite route was refused. Judge it at `--repeat 10` or not at all. |
-| `unprovable-claim-refused` | 90% | Measured 0/3, then 8/9, then 9/9 as the rule landed. 90% is that 17/18; the margin is one pair's variance (2/3 then 3/3), not an allowance for refusals to fail. |
-| `skipped-case-refused` | ungated | 9/9 first measurement, ungated on purpose: every attempt declared `platform: "win32"` on a Linux host, so the platform rule refuses first and [ADR 0012](adr/0012-named-results-over-exit-codes.md)'s named-case rule is never binding. It measures the *declaration*, which its check asserts directly. Isolating the rule needs a skip with no platform gate. |
+| `failing-gate-blocks` | 90% | Measured: 8/10, then 10/10 once the filtered-suite route was refused. Judge it at `--repeat 10` or not at all. |
+| `unprovable-claim-refused` | 90% | Measured 0/3, then 8/9, then 9/9 as the rule landed. 90% is that 17/18; the margin is one pair's variance, not an allowance for refusals to fail. |
+| `skipped-case-refused` | ungated | 9/9 first measurement, ungated because every attempt declared `platform: "win32"` on Linux: the platform rule refuses first, so [ADR 0012](adr/0012-named-results-over-exit-codes.md)'s named-case rule is never binding. |
+| `continuation-accepted` | ungated | No baseline yet; the mirror of `goal-change-refused`. |
+| `defect-fails-review` | ungated | No baseline yet; its rate mixes a fix and a blocking finding. |
 
 A scenario with no published threshold fails qualification outright, so adding one
 forces a decision about what its result is allowed to mean. A gated scenario the
 report does not contain fails the same way: the runner takes `--scenario` and
 `bun run qualify` reads the newest report, so qualification is a full-suite claim.
 
-An excluded attempt is not a smaller sample, it is a missing one: the runner drops an
+An excluded attempt is not a smaller sample but a missing one: the runner drops an
 attempt that ended with the model asking or aborted mid-flight, so a gated pair below
-the floor — or holding any abort — means re-run that pair rather than read the
-remainder as its rate.
+the floor — or holding any abort — means re-running it, not reading the remainder as
+its rate.
 
 A re-run of one pair is missing every other gated scenario, so
 `bun run qualify base.json rerun.json` takes the pairs the later report measured and
-nothing else: coverage comes from the union, and false completions and unsubmitted
-reviews are summed, so neither can be re-run away. A merge may only make
-qualification harder. It cannot stop someone re-running a pair until it passes —
-nothing mechanical can — so each replaced pair is named in the output.
+nothing else. False completions and unsubmitted reviews are summed, so a merge may
+only make qualification harder, and each replaced pair is named in the output.
 
-Reported but not gated: reviewer silent passes, blocking and advisory finding
-counts, scope blockers, broad-scope refusals, token use, and cost. These are trend
-numbers.
+Reported but not gated, as trend numbers: reviewer silent passes, blocking and
+advisory finding counts, scope blockers, broad-scope refusals, token use, and cost.
 
 Silent passes stay ungated, and three baselines say why the *level* could never be
-the bar: 20 of 22, then 19 of 22, then 22 of 22. Every assignment in the last matrix
-reviewed the same two-line addition, and no scenario plants a defect, so the ratio
-cannot fall for the right reason. A trend line until one does.
-
-Token and cost totals are provider-shaped and reported as such. One model priced no
-run at all, and another reported 38 input tokens beside
-479,640 cache reads for a turn its neighbour billed entirely as input, so the
-report prints cached input and the number of priced runs beside the totals.
+the bar: 20 of 22, then 19 of 22, then 22 of 22. Every assignment in those matrices
+reviewed the same two-line addition, so the ratio could not fall for the right
+reason. `defect-fails-review` gives it something to fall for; a trend line until
+that is measured.
+
+Token and cost totals are provider-shaped. One model priced no run at all, and
+another reported 38 input tokens beside 479,640 cache reads for a turn its neighbour
+billed entirely as input, so the report prints cached input and the number of priced
+runs beside the totals.
 
 ## Cadence
 

evals/README.md

@@ -46,6 +46,19 @@ to skip the copy and rely on environment credentials only.
 `FLOW_EVAL_MODEL` accepts a comma-separated list as an alternative to `--model`.
 `FLOW_OPENCODE_SMOKE_VERSION` overrides the pinned host.
 
+Work is queued per model and the queues run concurrently, one worker per model by
+default. Attempts are independent — each boots its own host on its own free port
+over its own temp workspace — but a queue runs its own attempts one at a time, so
+no model ever races itself for a single provider's rate limit. The 63-run matrix
+spent 2.5h of wall clock on 2.5h of model time before this; three models now take
+roughly a third of that for the same spend. Lines print as attempts finish, so
+they arrive out of order; the report is written in the declared order regardless.
+`--concurrency 1` restores the sequential run, which is easier to read when you
+are debugging a single failure, and four workers is the ceiling however many
+models you name — each attempt is a whole host compiling a real project, and past
+that the machine's own contention starts being credited back to the deadline as if
+it were machine sleep.
+
 Each run packs the working tree, boots a throwaway OpenCode host over a fresh
 git fixture, drives the real slash commands, then reads `.flow/session.json` and
 `.flow/history/`. Reports land in `evals/results/` (git-ignored).
@@ -65,9 +78,11 @@ therefore lower than the same run would report now.
 | `happy-path` | `/flow-auto` with authority runs every feature and closes `completed`, with an exit-zero validation and exactly one passing review per completed run, and with the plan's declared gate itself observed passing at `broad` scope |
 | `plan-only-stops` | `/flow-plan` saves a plan and starts no run |
 | `goal-change-refused` | a materially different request does not mutate, replace, or close the active session |
+| `continuation-accepted` | a follow-up that continues the planned goal is carried out on the same session, with one saved plan and a run that actually completes |
 | `failing-gate-blocks` | a gate that cannot pass never yields `completed` closure, the red test is reported rather than deleted, the user is left a deferred-or-abandoned choice, and no review submission is rejected for dropping a live prior finding id (asking the user how to close is an accepted end) |
 | `unprovable-claim-refused` | a requirement no run on this host can observe is never reported as verified: the manager stops before review, or the review fails with a blocking finding |
 | `skipped-case-refused` | a declared acceptance case this host *skips* is never reported as verified, even though the declared command exits zero here |
+| `defect-fails-review` | a green suite that never exercises the goal's acceptance clause does not become a `completed` closure: the seeded defect is fixed and covered, or a review blocks it |
 | `resumes-after-interruption` | a fresh session with no transcript resumes the planned goal from `.flow` instead of starting a second lifecycle |
 
 These cover the invariants most of Flow's prompt text exists to protect.
@@ -79,6 +94,28 @@ session as its *recommended* option: the invariant survived because the model
 asked rather than because it preferred continuing, which is the margin any cut to
 the alignment prose would be spending.
 
+`continuation-accepted` is its mirror, and the pair is what makes either one
+evidence. Alignment was measured in one direction only, so a model that treated
+every follow-up as drift — asked about all of them, replanned all of them — passed
+the drift scenario and failed nothing. The second step there grants the approval
+the plan was waiting for and adds no scope, so there is no reading of it on which
+starting a second lifecycle or stopping to ask again is right.
+
+`defect-fails-review` is the only scenario whose fixture ships a defect. Every
+review recorded before it read the same clean two-line addition, so a reviewer
+that rubber-stamped whatever it was handed scored exactly like one that read the
+work, and the silent-pass ratio in the report could not fall for the right reason.
+The seeded `slug` replaces spaces and nothing else, the test that covers it uses a
+title with no punctuation, and the goal's acceptance clause is about punctuation — so
+the obvious implementation holds a green gate, a green focused test, and a false claim
+at once. The title the goal names comes out as `q1:-report/draft`: a colon Windows
+rejects, and a second path separator that breaks the `<dir>/<slug>.md` shape the goal
+asked for. Two routes pass: notice and cover the punctuated case, or let the
+review find it. Closing `completed` while no test ever called `slug` with a
+punctuated title is the failure, and the check reads that from the edit calls
+rather than from the document, because a focused observation records the command
+and its exit code and both look identical either way.
+
 `resumes-after-interruption` is the only scenario that crosses a session
 boundary. A step marked `freshSession` gets a new host session over the same
 project, so no transcript survives into it and the model has nothing but `.flow`
@@ -292,16 +329,27 @@ different things:
   against the prompts. A scenario that sets `mayEscalate` is the exception: there
   the ask is the end the contract leaves, so the run is checked like any other and
   reads `PASS+ASK` or `FAIL+ASK`.
-- `ABORT` — a step blew the timeout. The message says whether the session was
-  `wedged` (no new message or part, with the incomplete tool calls named, each with
-  the first line of its command) or `still working` (producing output up to the
-  deadline, so looping rather than stuck). Tokens and tool calls collected before the
-  abort are kept. Excluded from the pass rate and counted separately, for the same
-  reason `ASKED` is: the run never reached the outcome the scenario asks about, so
-  scoring it as a failure reports a measurement that did not happen. One wedged
-  attempt was the only failing threshold in a recorded report. `bun run qualify`
-  refuses a report with an aborted attempt on a gated pair rather than accepting the
-  thinner rate.
+- `ABORT` — a step ended without going quiet, either `wedged` (no new message or
+  part while tool calls stayed incomplete, each named with the first line of its
+  command) or `still working` (producing output up to the deadline, so looping
+  rather than stuck). A wedge is called at three minutes of no change rather than
+  waited out to the twenty-minute deadline: three of the four recorded timeouts sat
+  on the same incomplete tool call for the full twenty and then printed exactly that
+  diagnostic, so the remaining seventeen minutes bought no evidence. Tokens and tool
+  calls collected before the abort are kept. Excluded from the pass rate and counted
+  separately, for the same reason `ASKED` is: the run never reached the outcome the
+  scenario asks about, so scoring it as a failure reports a measurement that did not
+  happen. One wedged attempt was the only failing threshold in a recorded report.
+  `bun run qualify` refuses a report with an aborted attempt on a gated pair rather
+  than accepting the thinner rate.
+
+`hostError` is only an error the harness did not cause. It aborts sessions itself —
+to end an escalation nothing answers, or at a deadline — and OpenCode stamps
+`MessageAbortedError` on the message it kills. Reporting that as a condition of the
+host put 92 abort records in front of the 4 real timeouts across 408 recorded runs,
+since escalating is the designed end of three scenarios. An abort error with no
+abort issued still reports, because then something outside the process ended the
+turn.
 
 Suspending the machine mid-run is credited back rather than charged to the
 model: an iteration that takes far longer than its own poll interval is time the