What
This repository is public and contains, for every scenario: the prompt, the scorer, the seed, and — increasingly — issues describing the answer in detail. An agent with network access can search for a scenario and read its own solution.
Nothing detects that today. skills.selfInstalled guards the analogous problem for skills (a product skill pulled into a -no-skills run invalidates that run as a baseline), and docs.calls records documentation fetches, but neither would notice a run that fetched github.com/hookdeck/evals.
Why now
outpost-003 asks an agent to configure operator events, and #34 publicly lists the exact route table needed to pass it. On 21 August a claude-code-sonnet-5 run passed that scenario, which was the trigger for checking.
That run was clean — verified from its tool calls:
hookdeck/evals: 0 occurrences, issues/34: 0 occurrences- zero
web_search / WebSearch calls of any kind
- its only
github.com fetch was raw.githubusercontent.com/hookdeck/outpost/main/docs/apis/openapi.yaml, which does not contain the routes
It found the endpoint by enumerating candidate paths, having first probed /totallybogus and /tenants/doesnotexist to learn what each kind of 404 looks like.
So this is a hole, not an incident. It is worth closing before it becomes one — the next scenario whose answer we write up in an issue is the one that gets read.
Proposed
A post-run check over recorded tool calls, flagging any run that references this repository — hookdeck/evals, github.com/hookdeck/evals, raw.githubusercontent equivalents, or a scenario id used as a search term.
Shape it like the self-installed-skill signal rather than a hard failure:
- record it on the result (
contamination or similar) rather than throwing, so the run is still scored and the evidence is kept
- surface it in
report-results, since a flagged cell cannot be read as a measurement either way
- exclude flagged cells from published snapshots, or publish them marked
Detection is cheap: tool calls are already stored per result, and this is a string search over them.
Worth deciding separately
Whether scenario prompts should be public at all. They are today, and the openness is deliberate — a published cell that nobody can inspect is not evidence. But the more of the answer we write down in issues, the more the benchmark documents its own solutions. A guard that detects reading is the cheap half; deciding what we publish is the other.
What
This repository is public and contains, for every scenario: the prompt, the scorer, the seed, and — increasingly — issues describing the answer in detail. An agent with network access can search for a scenario and read its own solution.
Nothing detects that today.
skills.selfInstalledguards the analogous problem for skills (a product skill pulled into a-no-skillsrun invalidates that run as a baseline), anddocs.callsrecords documentation fetches, but neither would notice a run that fetchedgithub.com/hookdeck/evals.Why now
outpost-003asks an agent to configure operator events, and #34 publicly lists the exact route table needed to pass it. On 21 August aclaude-code-sonnet-5run passed that scenario, which was the trigger for checking.That run was clean — verified from its tool calls:
hookdeck/evals: 0 occurrences,issues/34: 0 occurrencesweb_search/WebSearchcalls of any kindgithub.comfetch wasraw.githubusercontent.com/hookdeck/outpost/main/docs/apis/openapi.yaml, which does not contain the routesIt found the endpoint by enumerating candidate paths, having first probed
/totallybogusand/tenants/doesnotexistto learn what each kind of 404 looks like.So this is a hole, not an incident. It is worth closing before it becomes one — the next scenario whose answer we write up in an issue is the one that gets read.
Proposed
A post-run check over recorded tool calls, flagging any run that references this repository —
hookdeck/evals,github.com/hookdeck/evals, raw.githubusercontent equivalents, or a scenario id used as a search term.Shape it like the self-installed-skill signal rather than a hard failure:
contaminationor similar) rather than throwing, so the run is still scored and the evidence is keptreport-results, since a flagged cell cannot be read as a measurement either wayDetection is cheap: tool calls are already stored per result, and this is a string search over them.
Worth deciding separately
Whether scenario prompts should be public at all. They are today, and the openness is deliberate — a published cell that nobody can inspect is not evidence. But the more of the answer we write down in issues, the more the benchmark documents its own solutions. A guard that detects reading is the cheap half; deciding what we publish is the other.