docs(run-workflow): explain that {{KEY}} in run output is a redaction - #11
Merged
Merged
Conversation
Sim resolves a {{KEY}} reference at execution, then masks the value out
of the log-facing copy that `workflows runs get` and `logs get` return,
writing it back as the {{KEY}} label itself. Agents reading a run read
that as an unresolved reference and "fix" a workflow that already works
- rewriting {{KEY}} into environmentVariables.KEY, hardcoding a literal,
or printing the value to prove resolution.
The skill had nothing on this. It also cannot simply say "it's masked",
because an unknown name is passed through unchanged and renders the same
way, so the check has to key on the name rather than the rendering:
`secrets list` returns names plus an `unredacted` flag.
Also notes that a --follow stream is not a log copy and is not masked.
Both sides added a section to `## Diagnose failures`, so git flagged a textual conflict where there was no semantic one. Kept both. main's four run-record properties are an unheaded continuation of `## Diagnose failures`, so they stay attached to the numbered steps; the masking section follows as a peer H2, and the closing "Report which mode ran" paragraph stays last. No contradiction between the two: main's first bullet notes a completed run can sit in `redacting` status before its content is readable, which is the same pipeline the masking section describes once it is readable.
An adversarial review found the first draft wrong in the unsafe
direction on three counts, all verified in the sim source:
1. It said a listed name with `unredacted: false` is a mask, not a
failure. Unsound: MIN_SUBSTITUTABLE_LITERAL_LENGTH is 8, so a secret
shorter than 8 characters is never masked at all and its {{KEY}} is
ALWAYS an unresolved reference. `secrets list` proves a name exists,
never that it resolved in this run. The old line told an agent to
leave a genuinely broken reference in place.
2. Singling out `--follow` as "not a log copy" implied the default run
output IS masked. It is not - a plain run and a --follow stream both
POST the same executeWorkflow operation, and the execute route
contains no redaction call at all. Now says every live run response
is unmasked and only runs get / logs get return the masked copy.
3. "An unknown reference passes through unchanged" is wrong for one of
three languages: in a shell block an unresolved placeholder becomes
the empty string (shell.ts `resolved ? shellExpansion(...) : ''`),
while JavaScript and Python do leave it literal.
Net effect is a weaker but true claim: a {{KEY}} in a masked log is
usually a resolved secret, not proof of one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
{{KEY}}reference resolves to the real secret value at execution. Sim then builds a separate log-facing copy in which exact occurrences of that value are masked — and the mask it writes is the{{KEY}}label itself.That masked copy is what
workflows runs getandlogs getreturn. Coding agents driving the sim CLI read it, see{{OPENAI_API_KEY}}where a value should be, and conclude the reference never resolved. They then "fix" a workflow that already works: rewriting{{KEY}}intoenvironmentVariables.KEY, hardcoding a literal, or adding a print to prove resolution. This has been observed repeatedly in practice.No skill in the repo mentioned masking at all —
redact,masked, andscrubhad zero hits.Verified end to end against a live workflow before writing it up:
{{ANTHROPIC_API_KEY}}len: 108{{ANTHROPIC_API_KEY}}{{ZZ_DOES_NOT_EXIST_XYZ}}len: 25{{ZZ_DOES_NOT_EXIST_XYZ}}A resolved secret and a name that never existed render identically, both in a successful run.
Why the claim is hedged
An adversarial pass killed three stronger claims from the first draft, all checked against the sim source:
unredacted: falsedoes not mean "this is a mask."MIN_SUBSTITUTABLE_LITERAL_LENGTHis 8, so a secret shorter than 8 characters is never masked at all and its{{KEY}}is always an unresolved reference.secrets listproves a name exists, never that it resolved in this run. The original line would have told an agent to leave a genuinely broken reference in place.Singling out
--followimplied the default run output is masked. It is not — a plain run and a--followstream both POST the sameexecuteWorkflowoperation, and the execute route contains no redaction call. The section now says every live run response is unmasked and onlyruns get/logs getreturn the masked copy."Passes through unchanged" is wrong for one of three languages. In a shell block an unresolved placeholder becomes the empty string (
shell.ts,resolved ? shellExpansion(...) : ''); JavaScript and Python do leave it literal.Net effect is a weaker but true claim: a
{{KEY}}in a masked log is usually a resolved secret, not proof of one — and the section says what to check when it isn't.Scope
13 lines added to
skills/run-workflow/SKILL.md. Body only — the frontmatterdescriptionis unchanged, since this is about behaviour once the skill is routed to, not about routing.Merged
main(the new "four properties of runs and run records" section) and kept both: those bullets are an unheaded continuation of## Diagnose failures, so they stay attached to the numbered steps and the masking section follows as a peer H2. No contradiction — main's first bullet notes a completed run can sit inredactingstatus before its content is readable, which is the same pipeline this section describes once it is readable.bun run check:skills,bun run test, andbun run type-checkall pass.