Skip a scenario whose seed credentials are missing, rather than failing it - #54
Merged
Conversation
…ng it
`local/` files leave an unset `${VAR}` as written, so an agent is handed the
literal `${SEED_ACME_SQS_ACCESS_KEY}`. That reads as an obvious placeholder,
and an agent that notices is right to stop rather than configure delivery
that will silently fail. Measured on 24 August: exactly that scored 0/1
while three agents that did not notice scored 6/6.
Scoring it is the 13 August mistake again. A missing OUTPOST_API_KEY then
became six agent failures against named vendors, and the fix was to skip
rather than score. The same reasoning applies to a credential the scenario
supplies itself: a missing row is recoverable, a wrong row is not.
`unmetRequirements` now also reports any `SEED_*` placeholder a scenario's
workspace asks for and the environment cannot fill. Verified: without the
two variables, outpost-004 skips all six cells naming both; outpost-005,
which uses none, still plans its six.
The workflow passes the two secrets through so the scenario runs once they
exist. Until then CI skips it, which is why the weekly cron does not need
pausing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
local/files leave an unset${VAR}as written, so an agent is handed the literal${SEED_ACME_SQS_ACCESS_KEY}. That reads as an obvious placeholder — and an agent that notices is right to stop rather than configure delivery that will silently fail.Measured on 24 August: exactly that scored 0/1, while three agents that didn't notice scored 6/6.
This is the 13 August mistake in a new form
A missing
OUTPOST_API_KEYthen became six agent failures against named vendors, and the fix was to skip rather than score. The same reasoning applies to a credential the scenario supplies itself: a missing row is recoverable, a wrong row is not.unmetRequirementsnow also reports anySEED_*placeholder a scenario's workspace asks for and the environment cannot fill.Verified both directions: without the two variables
outpost-004skips all six cells naming both;outpost-005, which uses none, still plans its six.Why the weekly cron does not need pausing
CI has neither secret, and the workflow didn't pass them through even if it did. Before this change, Monday's run would have handed agents placeholders and published the failures. Now it skips the scenario cleanly.
The workflow passes both through, so
outpost-004starts running in CI as soon asSEED_ACME_SQS_ACCESS_KEYandSEED_ACME_SQS_SECRET_KEYare added as repository secrets. They are synthetic — any AKIA-shaped value works — and deliberately not committed, because a key realistic enough to convince an agent is realistic enough to trip push protection.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA