outpost-004: stop shipping AWS's documented example credentials - #52
Merged
Conversation
The workspace note carried AKIAIOSFODNN7EXAMPLE and its matching secret —
AWS's published placeholders, verbatim in most AWS tutorials ever written.
On 24 August claude-code-sonnet-5 recognised them. It had already derived
the whole solution — aws_sqs, config.queue_url, credentials as a separate
object, narrowing the webhook to retries — and stopped, because
configuring delivery with placeholder keys fails silently and the customer
receives nothing. It scored 0/1. Three other agents did not notice and
scored 6/6. The scenario was rewarding not checking your inputs.
A review flagged the credentials before the run and I judged the scoring
impact to be roughly zero. The run disagreed.
Replaced with `${ACME_SQS_ACCESS_KEY}` / `${ACME_SQS_SECRET_KEY}`, expanded
from the environment on the way into the workspace — the mechanism
HOOKDECK_WEBHOOK_SECRET already uses. Committing a realistic literal is not
an option: the first attempt was rejected by GitHub push protection, which
is the whole problem in miniature. A key convincing enough to fool an agent
is convincing enough to look like a leak.
Values are synthetic and live in .env; .env.example documents them. CI needs
the two variables added before this scenario runs there — without them the
note shows the literal placeholders, which reads as missing rather than as
configured, and is the documented behaviour of the expansion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
`local/` files expand `${VAR}` from the session's sandboxEnv, not from the
host environment — and sandboxEnv was a fixed list of three Hookdeck
variables. So a scenario needing a credential of its own had no way to
supply one without editing runtime.ts.
Measured: outpost-004 put `${ACME_SQS_ACCESS_KEY}` in its workspace note
with the value set in .env, and the agent was handed the literal
placeholder. It read exactly like the AWS documentation keys it had just
replaced — obviously fake — and the agent stopped rather than configure
delivery it believed would fail. Same failure, new cause.
Any `SEED_*` variable in the environment now reaches sandboxEnv. A prefix
rather than a wholesale passthrough: a workspace file saying
`${OPENAI_API_KEY}` should not be able to help itself to one, so exposure
is opt-in by name.
outpost-004's variables are now SEED_ACME_SQS_ACCESS_KEY and
SEED_ACME_SQS_SECRET_KEY. Verified they arrive in sandboxEnv with the
expected shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workspace note carried
AKIAIOSFODNN7EXAMPLEand its matching secret — AWS's published documentation placeholders, verbatim in most AWS tutorials ever written.What it did to a run
claude-code-sonnet-5recognised them, having already derived the whole solution —aws_sqs,config.queue_url, credentials as a separate object, narrowing the webhook toretries— and stopped, because configuring delivery with placeholder keys fails silently and the customer receives nothing.It scored 0/1. Three other agents didn't notice and scored 6/6. The scenario rewarded not checking your inputs — the inversion AGENTS.md names as the worst thing a scorer can do. A review flagged these credentials before the run and I judged the impact "roughly zero"; the run disagreed.
Three attempts, three different failures
${VAR}placeholders — the documented mechanism, but expansion reads fromsandboxEnv, which was a fixed list of three Hookdeck variables. The agent was handed the literal${ACME_SQS_ACCESS_KEY}, which reads exactly as fake as the AWS keys, and stopped for the same reason.The fix
Any
SEED_*variable in the environment now reachessandboxEnv, so a scenario can supply a credential of its own without editing the runtime. A prefix rather than a wholesale passthrough: a workspace file saying${OPENAI_API_KEY}should not be able to help itself to one.Verified end to end — the literal placeholder is gone from the transcript, the values reached the agent, and the existing redaction layer caught them on the way back out (
<redacted:SEED_ACME_SQS_ACCESS_KEY>).Result:
004now passes 6/6 across every arm, including the Sonnet cell that previously scored 0/1.CI needs
SEED_ACME_SQS_ACCESS_KEYandSEED_ACME_SQS_SECRET_KEYas repository secrets before this scenario runs there.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA