Correct three things the Outpost runs showed were wrong - #44
Merged
Conversation
The +skills experiments load ['hookdeck', 'event-gateway'], so an Outpost scenario got a router plus the wrong product's skill — event-gateway's only Outpost content is one line telling the agent to go elsewhere. The outpost skill, 39 files of it, had never been used by a run. "+skills" should mean the relevant skill is loaded. Whether an agent can find its way to the right skill from the router alone is a different question and deserves its own scenario, not to be smuggled into four. Adds `extra_skills`, which adds to an experiment's list only when that experiment has skills at all. Neither alternative works: - `skills` replaces the experiment's list outright, including for a -no-skills experiment, which would hand the baseline arm the very skill it exists to do without. - Adding outpost to the experiments themselves changes the skill set for all seventeen scenarios, making every published +skills row non-comparable to answer a question about four of them. Verified per arm: an Outpost scenario resolves to ['hookdeck','event-gateway','outpost'] with skills and [] without; a non-Outpost scenario is unchanged. This changes what outpost-001 measures, so it is re-run with the new three rather than left mixing methodologies. Its published passes came from agents self-installing the outpost skill over the network — both passing rows carry selfInstalled: ['outpost'], and no row passed without it. That finding is what prompted this: the skill was doing the work, by a route we had not designed and that depends on the registry being reachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
The prompt addendum's job is disclosure — it already says a project exists and how it is authenticated. It was silently omitting a second project that the harness injects whenever a scenario needs Outpost, while asserting "so both the CLI and the REST API are available to you", which reads as *this is your access*. That is not a discovery test we designed. Measured across twelve baseline cells on four Outpost scenarios: OUTPOST_API_KEY was used as a credential exactly zero times and every cell failed. Nine of the twelve agents believed they had succeeded, having built the task on api.hookdeck.com. Two reported with authority that the harness had given them the wrong credential, while a working key sat unread in their environment. The skill was the only artefact in the sandbox naming the variable, so the skills delta could not be separated from credential disclosure — the run measured our own omission as if it were a property of the skill. One credential type, two projects. Both keys authenticate api.hookdeck.com and the CLI; only the Outpost project's key reaches the Outpost subdomain, where another project's key gets a 404 that does not say why (#39). A developer using Outpost knows they use it and has the key in their environment; nobody learns their own credentials by enumerating env vars. The addendum names the project, the variable, and that the API has its own subdomain. It does not give the subdomain, the routes or the destination types, so how Outpost works is still what the scenarios measure. Scoped to the pinned CLI: against 2.5.0 the Outpost key authenticates the CLI and selects the project but has nothing useful to do there, so the API is the honest route to name. 3.0.0-beta.1 is published and adds managing an Outpost project from the CLI; when that pin moves this becomes incomplete rather than wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
**outpost-003 predicted universal failure on a scenario nobody fails.** The comment said "expect every agent to fail this initially, and publish it anyway", reasoning that an agent has no source of truth except what it can read. Agents do not work that way. It has now passed six times out of six across three models and both arms, including a weak model with no skills, by enumerating candidate routes — one first probed a deliberately bogus path to learn what a real 404 looks like, which is the control I failed to run when I concluded the API did not exist. So it measures persistence rather than discovery, does not currently discriminate, and is high variance: the same model passed 4/4 in one run and failed 0/1 in the next. #34 stands regardless — an endpoint reachable only by guesswork is undocumented whether or not an agent gets there. **outpost-002 counted any three successes on the tenant.** Three fresh publishes satisfied it as well as recovering the outage, and it passed an agent that reported retrying 54 events out of 78 found, because events outlive the tenant and a shared project accumulates them. Now counted per event: an event with a failed attempt is one the customer missed, and it is recovered when that same event_id also has a success. Republishing cannot fake it, since a new event has no failed attempt. **outpost-004 gave a do-nothing run 2/6.** Both negative checks are satisfied by the untouched seed, so no work scored two marks — the shape a crashed cell wears, and one wore it on 21 August before being spotted. The verdict was never wrong, since passed is the conjunction; the per-check count misled anyone reading detail, including us triaging a run. A run that changed nothing now returns a single failing check that says so. Verified against the live API: outpost-002 2/2 solved and fails unsolved; outpost-004 2/2 solved and a do-nothing run reports 0/1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
…o far The four existing Outpost scenarios barely discriminate. In a 24-cell run +skills went 12/12 and two scenarios were 6/6, because their difficulty had quietly been "work out that Outpost is the product" and the harness stopped withholding that. What was left is setup, and setup is not where agents fail. This is built the other way round, from the shape AGENTS.md says discriminates: an agent can finish, report success, and be wrong, with nothing erroring. Acme receive three topics and ask us to stop one. The cheapest fix — set their destination to the topic the ticket talks about — stops the cancellations they complained about and silently stops order.shipped, which they never mentioned because it was working. No error, no failed delivery; they notice days later when something they depend on has stopped arriving. Verified all three paths against the live API: correct fix 5/5 pass naive fix fails only "still receives the shipping events they depend on" do nothing fails only "no longer receives order cancellations" Two wrong answers, failing on opposite checks, neither raising an error. Scored on behaviour rather than configuration: it publishes real events of each topic and checks what arrives, so an agent that scopes the tenant instead of the destination passes, and config that merely looks right fails. Not built, deliberately: the better trap is changing deployment-level TOPICS, which fixes the complainant and silently breaks every other customer — an agent did exactly that to this project on 21 August. A scenario rewarding that would break every subsequent cell in a run until #41 persists a config baseline. Worth revisiting then. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
AGENTS.md already says to read the agent's report first and calls it the cheapest answer every time. It keeps getting skipped, including by whoever wrote that line. An exhortation loses to a scoreboard, because the scoreboard is right there and the transcripts are not. The case is stronger than "sometimes useful": over two days of Outpost runs every product finding came out of a transcript and none came out of the scoreboard — an API answering 404 where 401 was meant, a skill that never names the credential, docs whose env vars do not exist, and a harness omission of ours that failed twelve baseline cells and read as a skills result. A pass rate cannot express any of those, and cannot separate "the agent could not" from "we misled it". So this turns the convention into a list, from fields the harness already records: unclean exit, failed-while-claiming-success, a declared credential never referenced, a skill offered and never opened, a baseline that self-installed a product skill, and a cell whose only green checks are the ones an idle agent satisfies. Three false positives were removed before it was worth trusting, because a triage tool that cries wolf returns us to nobody reading transcripts: - required credentials now come from each scenario's `requires`, not a blanket flag, which had faulted the ElevenLabs and Stripe scenarios for not using an Outpost key - self-install only flags a *product* skill in a *baseline* arm; a provider skill like stripe-webhooks is legitimate and AGENTS.md says so - "only negatives passed" now requires every green check to be negative, not any: an agent that re-enabled a destination but never recovered the held events did half the job, and calling that idle misdescribes it Against the current results it flags two cells: the one that produced findings #39 and #40, and a failed capability question answered with no tool calls — which is that scenario's whole point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
Three faults, found by auditing the scorers rather than by a run. **It threw on the failures it exists to catch.** The sanity guard fired whenever acme received nothing — which is exactly what deleting the destination, disabling it, or emptying its topics produces. The checks array describing those failures was built and then discarded, the throw propagated, and no result row was written at all. checkNothingDisabled existed to name a route the scorer then threw away. Now only a platform fault throws: acme has an enabled destination subscribed to the topic and still received nothing. If nothing arrived because no destination subscribes any more, the agent did that and the checks say so. Verified: disable now scores 2/5 naming the switched-off destination, delete scores 1/5, and empty topics is rejected by the API. **A vanished tenant crashed the read.** Deleting a tenant's last destination deletes the tenant, so every later read 404s. `list` now treats 404 as an empty read — and only 404, because swallowing everything reports a confident agent failure for a platform blip. **The collateral check failed on the runs where it mattered.** Globex shared one publish with acme, so its event was 45 seconds old whenever acme's wait ran to timeout — precisely when the agent had broken acme. Measured: globex passed with a correct fix and failed twice on the delete and disable paths, while delivering fine when tested alone. The exact interaction was never pinned down and this does not guess at it; globex now gets its own publish and its own window, so acme's timing cannot decide whether globex looks untouched. Accusing an agent of breaking a customer it never touched is the worst false failure here. Also: the header justified behaviour-scoring with a lever that does not exist — it claimed an agent could edit a tenant's topic list. Tenant topics are read-only and derived from destinations, verified live: a PUT with topics returns [], and adding a destination on order.shipped makes it ["order.shipped"]. The tenants[].topics field in every seed here is inert. The real reason is per-destination filters, which config-shaped scoring would miss. Polling widened from 1s to 3s: two back-to-back waits at one second each issue well over a hundred requests per cell, and waitFor treats a failed probe as "not ready", so a rate-limited read is indistinguishable from nothing arriving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This was the only Outpost scenario with no seed, so nothing removed
`acme` before the agent ran — and every other Outpost scenario seeds a
tenant called `acme` carrying a destination on an order-ish topic. Tenant
cleanup runs on release inside a catch-and-ignore and is skipped when a
run is killed, both of which have happened. A survivor satisfied all three
checks with no agent action and published green against a named vendor.
project-source.ts argues the residual leak is tolerable because it makes
this scenario's first check pass visibly. That has it backwards: it makes
every check pass, and nothing about the row looks wrong.
Fixed in the seed, with a new `deleteTenants`, because the state has to be
absent rather than distinguishable. Scoping by `created_at >= acquiredAt`
was tried first and fails the correct answer: tenant create is idempotent,
so an agent that properly PUTs an existing id gets the original timestamp
back and is scored as inheriting a leftover. Measured — the tenant read two
minutes older than the lease about to score it.
Two more faults in the same file:
- "an order event reaches the customer" counted attempts of any status, so
a destination pointed at an invented hostname passed while the customer
received nothing. Now counts successful attempts. outpost-004's header
claims delivery is "already proven by outpost-001"; it was not.
- listDestinations read `{ data }` alone — the trap this file's own
comments warn about twice — surviving only because that endpoint is
unpaged and returns a bare array.
The residual, called out in the code rather than fixed quietly: the ticket
never says where the customer's endpoint is, so any reachable URL passes.
One agent stood up a localtunnel inside its own sandbox, delivered to
itself, passed, and offered to "swap the temporary receiver URL for the
real customer endpoint next". Fixing that means giving the ticket an
endpoint, which changes a published scenario.
Adds the SOLUTION.ts this scenario never had, which is why score-only
could only ever exercise its failing path — and how the created_at
mistake was caught before it shipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This was referenced Aug 24, 2026
**The addendum was gated on the machine, not the scenario.** `outpostClient` is built whenever OUTPOST_API_KEY is set, which is always, so every Event Gateway scenario was being told about an Outpost project it has no use for. That is noise in fourteen prompts to answer a question about five, and it would have made every published cell non-comparable rather than just the Outpost ones — which is the difference between a patch release and a full re-run. Now gated on the scenario's seed declaring Outpost state; verified that outpost-002 sees the sentence and filtering-001 does not. **outpost-005's ticket never set its trap.** The scorer described the failure as "set their destination to the topic they still talk about", but the ticket named no topic acme wanted — only the one to remove — so narrowing was not a candidate action and the wrong answer took more work than the right one. It also said "everything else they get today should carry on exactly as it is, and Globex shouldn't be affected at all", which states four of the five checks and replaces the silence the scenario depends on with an instruction to enumerate and preserve. All six agents passed. The ticket now names order confirmations as working and relied upon, says nothing about shipping events, and does not mention Globex. Verified: the naive fix — scoping to the named keeper — scores 4/5, failing only "acme still receives the shipping events they depend on". The lesson is in the header: verifying that a scorer rejects a hand-written wrong answer proves the scorer works, not that agents make that mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
`v5.ts` and `t5b.ts` are throwaway verification scripts. They have to live inside apps/framework to resolve its imports, so they are not covered by the temp-directory convention, and both were swept in by `git add -A` after the command that would have deleted them timed out. CI caught the second one on formatting; the first had been sitting in the branch for two commits because it happened to be formatted correctly. Scratch files are now `*.probe.ts` and gitignored, so the next one cannot be added by accident rather than relying on remembering to delete it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three corrections, all found by running the scenarios rather than by review.
outpost-003predicted universal failure on a scenario nobody failsThe scorer said "Expect every agent to fail this initially, and publish it anyway", reasoning that an agent has no source of truth except what it can read.
It has now passed six times out of six across three models and both arms — including a weak model with no skills at all. Agents find the undocumented route by enumerating candidates (
/operator-events,/operator-event-destinations,/operator/destinations…). One first probed a deliberately bogus path to learn what a real 404 looks like, which is precisely the control I failed to run when I concluded the API didn't exist.So it measures persistence, not discovery, doesn't currently discriminate, and is high variance — the same model passed 4/4 in one run and failed 0/1 in the next, because passing turns on which paths get guessed. A single attempt publishes a coin flip.
#34 stands regardless: an endpoint reachable only by guesswork is undocumented whether or not a determined agent gets there.
outpost-002counted any three successes on the tenantThree fresh publishes satisfied it as well as recovering the outage. It passed an agent that reported retrying 54 events out of 78 it had found — events outlive the tenant, and a shared project accumulates them.
Now counted per event: an event with a failed attempt is one the customer missed, and it's recovered when that same
event_idalso has a success. Republishing can't fake it, because a new event has no failed attempt. No timestamps needed.outpost-004gave a do-nothing run 2/6Both negative checks — retries still reaching the old endpoint, other customer untouched — are satisfied by the untouched seed. So a run that did nothing scored two marks. That's the shape a crashed cell wears, and one wore it on 21 August before being spotted.
The verdict was never wrong (
passedis the conjunction, and the four positive checks need real work). The per-check count misled anyone reading detail — including us, triaging a run. A run that changed nothing now returns a single failing check saying so.Verification
Against the live API:
outpost-002: 2/2 withSOLUTION.ts, fails unsolvedoutpost-004: 2/2 withSOLUTION.ts; do-nothing run reports 0/1, not 2/6Context
These came out of a 24-cell run whose headline changed completely once the harness stopped omitting the Outpost project from its environment description:
+skills 12/12,-no-skills 10/12, against a previous run's 9–0. Roughly seven of those nine cells were measuring our own defect.Follow-up work is to make the scenarios discriminate again — all four were designed when product identification was silently supplying the difficulty.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA