What happened
Until #44 the harness prompt addendum named only the Hookdeck project — "HOOKDECK_API_KEY is set in your environment, so both the CLI and the REST API are available to you" — while silently injecting OUTPOST_API_KEY for scenarios that need it. The +skills arm meanwhile loaded hookdeck and event-gateway: a router plus the wrong product's skill, whose only Outpost content is a line telling the agent to go elsewhere.
The skill's reference files were therefore the only artefact in the sandbox naming OUTPOST_API_KEY.
What it measured
Across twelve baseline cells on four Outpost scenarios, on 21 August:
OUTPOST_API_KEY was used as a credential zero times- every cell failed
- nine of twelve agents reported success, having built the task on
api.hookdeck.com
That produced a 9–0 Outpost skills delta. After the addendum was corrected, the same 24 cells read +skills 12/12, -no-skills 10/12 — a delta of 2 cells in 24, in different models on different scenarios.
So roughly seven of the original nine cells were measuring our own omission, and it was hours from being published as evidence that skills work.
Why this is filed rather than left in the PR
AGENTS.md: "A finding that changes what someone should do belongs in an issue." This changes how every future skills delta should be read — assume most of an apparent skills win is our own instrument until proven otherwise — and it belongs in LOOPS.md as a measurement that measured the instrument rather than the product.
If the PR is squashed, the 2-in-24 result exists nowhere else.
Follow-on
The +skills arm now receives the outpost skill via extra_skills, which makes #26 (silently drifting submodule pin) more consequential, not less.
What happened
Until #44 the harness prompt addendum named only the Hookdeck project — "HOOKDECK_API_KEY is set in your environment, so both the CLI and the REST API are available to you" — while silently injecting
OUTPOST_API_KEYfor scenarios that need it. The+skillsarm meanwhile loadedhookdeckandevent-gateway: a router plus the wrong product's skill, whose only Outpost content is a line telling the agent to go elsewhere.The skill's reference files were therefore the only artefact in the sandbox naming
OUTPOST_API_KEY.What it measured
Across twelve baseline cells on four Outpost scenarios, on 21 August:
OUTPOST_API_KEYwas used as a credential zero timesapi.hookdeck.comThat produced a 9–0 Outpost skills delta. After the addendum was corrected, the same 24 cells read
+skills12/12,-no-skills10/12 — a delta of 2 cells in 24, in different models on different scenarios.So roughly seven of the original nine cells were measuring our own omission, and it was hours from being published as evidence that skills work.
Why this is filed rather than left in the PR
AGENTS.md: "A finding that changes what someone should do belongs in an issue." This changes how every future skills delta should be read — assume most of an apparent skills win is our own instrument until proven otherwise — and it belongs inLOOPS.mdas a measurement that measured the instrument rather than the product.If the PR is squashed, the 2-in-24 result exists nowhere else.
Follow-on
The
+skillsarm now receives theoutpostskill viaextra_skills, which makes #26 (silently drifting submodule pin) more consequential, not less.