Four of five Outpost scenarios do not discriminate

[leggetter]

What

With the harness no longer withholding that Outpost is the product (#46), the Outpost suite barely separates agents. From the corrected 24-cell run plus a 6-cell run of outpost-005:

scenario	result	discriminates
001 customer subscriptions	6/6	no
002 disabled destination	5/6	yes — weak model skips recovering the held events
003 operator events	6/6	no
004 queue destination	5/6	yes — one model builds it in the wrong product
005 topic scoping	6/6	no

+skills passes everything.

003 is worse than flat — it is variance

It was written predicting universal failure ("Expect every agent to fail this initially"). It has passed 6/6 across three models and both arms, including a weak model with no skills, by enumerating candidate routes until one answers. And the same model passed 4/4 in one run and 0/1 in the next, because passing turns on which paths get guessed. A single attempt publishes a coin flip.

005 was designed to discriminate and does not

Its trap — narrowing a customer's topics too far — was verified against a hand-written wrong answer, which proves the scorer works and says nothing about whether agents make that mistake. They do not: all six read the current topic list and removed one element. The prompt also stated the invariant outright ("everything else they get today should carry on exactly as it is"), and the wrong answer took more work than the right one.

The lesson worth keeping

Both scenarios that do discriminate were found by accident, not design — 004 was built expecting a judgement trap and caught product confusion instead. Designing hard scenarios from an armchair has not worked. The productive route looks like mining transcripts for mistakes agents actually made, which triage.ts now surfaces.

Done when

Each of 001, 003, 005 either discriminates or is recorded as a deliberate floor, and 003 is given multiple attempts or held back until #34 makes its route findable.

[leggetter]

Updated after v0.3.0. Four of five, not three — outpost-005 was rewritten to set its trap properly and still went 6/6.

scenario	v0.3.0	discriminates
001 customer subscriptions	5/6	one weak-model cell built it in the Event Gateway
002 disabled destination	5/6	one weak-model cell asked for confirmation and stopped
003 operator events	6/6 (majority of 3)	no
004 queue destination	6/6	no
005 topic scoping	6/6	no

The "coin flip" claim was too strong

003 ran three full passes, 18 observations. 16 passed. The frontier models were stable across all six of their observations; only the weak model split, 2–1 in both arms. So it is not a coin flip — it is stable at the top and unreliable at the bottom, which is the ordinary shape of a floor.

Its two failures were different mistakes, and the second is the interesting one: one agent built Hookdeck's Event Gateway notification webhooks instead of Outpost operator events — documented, adjacent, and wrong — while the other spent 165 tool calls enumerating undocumented routes and timed out. When the correct API is undocumented and an adjacent one is well documented, a weaker agent reliably finds the documented wrong thing. That belongs to #34.

Why 005 still fails to discriminate

Its ticket now names a topic the customer wants, so narrowing to it is a live wrong answer. Every agent still passed, because the API shape defeats the trap: PATCH takes the whole topics array, so the destination read an agent must do anyway returns all three topics and removing one element preserves the rest by construction. The wrong answer takes more work than the right one.

Making it discriminate would mean the true received set being something an agent cannot get from the read it is already doing — spread across destinations, or visible only in delivery history. That is a redesign, not a prompt edit.

What this suggests

Both scenarios that do discriminate were found by accident: 004 was built expecting a judgement trap and caught product confusion instead. Three deliberate attempts at designing a hard scenario have produced three floors. The route that has worked is mining transcripts for mistakes agents actually made, which triage now surfaces.