Publish the Outpost snapshot: 104/114

[leggetter]

Adds the four new Outpost scenarios and re-measures outpost-001, whose scorer changed materially. 114 benchmark cells across 19 scenarios; 30 are Outpost and 28 pass.

What was re-run, and what carried over

	cells
Event Gateway	84	carried over from v0.2.0 untouched
Outpost	30	freshly run

Carrying the 84 over is only legitimate because the prompt addendum is now gated on the scenario rather than on the machine having an Outpost key. Before that fix every prompt in the suite would have changed and no old row would be comparable.

outpost-003 is a majority of three passes

Worth the extra 12 cells. Both weak-model cells split 2–1, and the two failures were different mistakes:

• one built Event Gateway notification webhooks instead of Outpost operator events — documented, adjacent, wrong
• the other timed out at 165 tool calls, still enumerating undocumented routes

A single attempt would have published either as a capability gap. The frontier models were stable across all six observations. Published rows are pass 3, which matched the majority.

That second failure sharpens #34: when the correct API is undocumented and an adjacent one is well documented, a weaker agent reliably finds the documented wrong thing.

compare-snapshots was broken and blamed the wrong thing

Snapshots are ~9.6 MB. execFileSync defaults to a 1 MB buffer, so reading a tagged snapshot failed with ENOBUFS, and this file's own catch reported:

could not read results/latest.json at v0.2.0. Is it a tag or commit in this repository?

…sending you to check a tag that was present and readable. The one tool the release process has for not claiming progress that didn't happen had quietly stopped working at the size where it becomes useful.

With it fixed it does its job, refusing to credit three outpost-001 flips to the agents because that scenario's scorer, prompt and seed all changed:

Changed where scoring also changed (3) — NOT attributable to the agent:
  fail -> pass  outpost-001 x claude-code-sonnet-5-no-skills
  fail -> pass  outpost-001 x codex-gpt-5.4-mini
  fail -> pass  outpost-001 x codex-gpt-5.6-no-skills

What this snapshot does not show

Every Outpost scenario is at 6/6 by majority except the two single failures above. There is no skills signal: +skills and -no-skills are within one cell of each other, and one of the two skills-arm failures was an agent asking for confirmation rather than failing the task. The release should claim coverage, not a skills result.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA

[leggetter]

Draft release notes for v0.3.0

Title: Outpost coverage goes from one scenario to five

---Outpost had one scenario in this benchmark. It now has five, covering customer
subscriptions, a destination switched off after repeated failures, alerting, delivery to
a queue, and narrowing what one customer receives.

Running them turned up four gaps, all in what an agent can read rather than in what the
product does. The API that configures Outpost's alerts appears in no spec, no reference
and no skill. Every environment variable on the operator events documentation page is
wrong. A key belonging to the wrong project type is reported as "Not Found" rather than
as the wrong key, which sent one agent to build the whole task in the wrong product. And
the Outpost skill never says which key you need.

Nothing shipped to the product this cycle. The four gaps are open and none is ours to
close inside this repository.

Scenarios: 19 (+4)

Snapshot: 114 rows · 6 experiments · 19 scenarios · 104 passed.

The 84 Event Gateway rows carry over unchanged from v0.2.0. The 30 Outpost rows are new
or re-measured, so totals either side of this release cover different work and should not
be subtracted — the per-cell comparison is under "How the numbers moved".

Shipped

Nothing. The four findings below are open against the docs, the API and the skill, and
none of them is ours to close inside this repository.

Benchmark

• Outpost coverage went from one scenario to five · Product coverage: Console still has no scenarios #19
• A scenario could be passed entirely by a tenant left behind by an earlier run · Correct three things the Outpost runs showed were wrong #44
• "The customer received it" counted a delivery attempt, so a destination pointed at an invented host passed · Correct three things the Outpost runs showed were wrong #44
• A scenario threw away the checks describing its own worst failures instead of scoring them · Correct three things the Outpost runs showed were wrong #44
• A scenario failed an agent for refusing AWS's documented placeholder credentials · outpost-004: stop shipping AWS's documented example credentials #52
• A scenario supplies its own credentials without committing them · outpost-004: stop shipping AWS's documented example credentials #52
• Which cells of a run are worth reading is now a command, not a convention · Correct three things the Outpost runs showed were wrong #44
• Every scenario records whether a failure is ours to fix or the model's · Correct three things the Outpost runs showed were wrong #44
• The tool that stops us claiming progress that did not happen had silently stopped working · Publish the Outpost snapshot: 104/114 #53
• The harness never told agents about the Outpost project they had been given · The prompt addendum omitted the Outpost project, and twelve baseline cells were read as a skills result #46

Discovered

• Turning on alerts for a disabled destination needs an API that appears in no spec or doc · The operator events API is undocumented: absent from both OpenAPI definitions and from the docs #34
• Every environment variable on the operator events documentation page is wrong · Outpost docs give the wrong env var name for every operator events setting #32
• A key for the wrong project type is reported as "Not Found" rather than as the wrong key · The Outpost API answers "Not Found" when your key belongs to a non-Outpost project #39
• The Outpost skill never says you need a key belonging to an Outpost project · The Outpost skill never says you need a key for an Outpost project #40

How the numbers moved

Against v0.2.0, on the 90 cells present in both:

Changed where scoring also changed (3) — NOT attributable to the agent:
  fail -> pass  outpost-001-customer-subscriptions x claude-code-sonnet-5-no-skills
  fail -> pass  outpost-001-customer-subscriptions x codex-gpt-5.4-mini
  fail -> pass  outpost-001-customer-subscriptions x codex-gpt-5.6-no-skills

Those three moved because that scenario's scorer, prompt and seed were all corrected —
no agent behaved differently. Everything else on the comparable set is unchanged, and
24 cells are new.

outpost-003 is published from a majority of three passes rather than one. It was worth
the extra twelve cells: both weak-model cells split 2–1, and the two failures were
different mistakes — one built Hookdeck's Event Gateway notification webhooks instead of
Outpost operator events, the other spent 165 tool calls enumerating undocumented routes
and ran out of time. The frontier models were stable across all six observations. A single
attempt would have published either failure as a capability gap.

What this release does not claim

There is no skills result here. Across the Outpost scenarios, +skills and -no-skills
land within one cell of each other, and one of the two skills-arm failures was an agent
asking for confirmation rather than one that could not do the task.

An earlier run of the same scenarios read 9–0 in favour of skills. That number was our own
harness omitting a credential from the prompt while injecting it into the environment;
the skill was the only artefact in the sandbox that named it. The corrected measurement is
2 cells in 24, and the whole episode is recorded as Loop 2 in LOOPS.md.

Four of the five Outpost scenarios are passed by every agent. That is an honest floor and
roughly the ratio upstream reports, but it means this coverage does not yet separate
agents from one another. #47 tracks it.