What happened
On 24 August the scheduled eval-refresh fired at 06:14 UTC against main and was cancelled by hand at roughly 47 of 72 cells, minutes ahead of publish-results.
Had it completed it would have published:
- Outpost cells failing because of our own harness omission. The prompt addendum on
main still names only the Hookdeck project while injecting OUTPOST_API_KEY silently (#46). Twelve baseline cells failed that way on 21 August, nine of them reporting success against the wrong product.
outpost-001 results a leftover tenant can produce, and a delivery check that counts attempts rather than deliveries. Both fixed in #44, neither on main.004 and 002 collateral checks that cannot detect the failures they describe (#48).
The fixes existed, reviewed, on an open PR. The cron neither knows nor can be told.
The gap
publish-results is gated on the matrix succeeding and nothing else. There is no way to express "the instrument is known-broken this week, run but do not publish" short of cancelling the run by hand — which requires somebody to be watching at the right moment. Nobody was; it came up in conversation by luck.
This is adjacent to but distinct from #41 and #45, which are about individual cells being wrong. This is about a whole snapshot being published while we already know it is wrong.
Options worth weighing
- A
publish: false switch — a repo variable or a marker file — checked by publish-results, so a run still produces evidence without updating the contract.
- Publish only from a tagged commit rather than from
main, which matches how the page already reads results (it resolves the latest release tag) and would make publishing a deliberate act.
- A staleness gate: refuse to publish if
main has scenario or scorer changes newer than the last release, on the grounds that the suite has moved and the numbers are not comparable.
The second is closest to the release discipline in AGENTS.md — "a release represents one run and what changed since the last one" — and would have prevented this without anyone watching.
What happened
On 24 August the scheduled
eval-refreshfired at 06:14 UTC againstmainand was cancelled by hand at roughly 47 of 72 cells, minutes ahead ofpublish-results.Had it completed it would have published:
mainstill names only the Hookdeck project while injectingOUTPOST_API_KEYsilently (#46). Twelve baseline cells failed that way on 21 August, nine of them reporting success against the wrong product.outpost-001results a leftover tenant can produce, and a delivery check that counts attempts rather than deliveries. Both fixed in #44, neither onmain.004and002collateral checks that cannot detect the failures they describe (#48).The fixes existed, reviewed, on an open PR. The cron neither knows nor can be told.
The gap
publish-resultsis gated on the matrix succeeding and nothing else. There is no way to express "the instrument is known-broken this week, run but do not publish" short of cancelling the run by hand — which requires somebody to be watching at the right moment. Nobody was; it came up in conversation by luck.This is adjacent to but distinct from #41 and #45, which are about individual cells being wrong. This is about a whole snapshot being published while we already know it is wrong.
Options worth weighing
publish: falseswitch — a repo variable or a marker file — checked bypublish-results, so a run still produces evidence without updating the contract.main, which matches how the page already reads results (it resolves the latest release tag) and would make publishing a deliberate act.mainhas scenario or scorer changes newer than the last release, on the grounds that the suite has moved and the numbers are not comparable.The second is closest to the release discipline in
AGENTS.md— "a release represents one run and what changed since the last one" — and would have prevented this without anyone watching.