Skip to content

docs(run-workflow): explain that {{KEY}} in run output is a redaction - #11

Merged
mzxchandra merged 3 commits into
mainfrom
docs/run-workflow-secret-masking
Sep 15, 2026
Merged

mzxchandra merged 3 commits into
mainfrom
docs/run-workflow-secret-masking

Conversation

@mzxchandra

@mzxchandra mzxchandra commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Problem

A {{KEY}} reference resolves to the real secret value at execution. Sim then builds a separate log-facing copy in which exact occurrences of that value are masked — and the mask it writes is the {{KEY}} label itself.

That masked copy is what workflows runs get and logs get return. Coding agents driving the sim CLI read it, see {{OPENAI_API_KEY}} where a value should be, and conclude the reference never resolved. They then "fix" a workflow that already works: rewriting {{KEY}} into environmentVariables.KEY, hardcoding a literal, or adding a print to prove resolution. This has been observed repeatedly in practice.

No skill in the repo mentioned masking at all — redact, masked, and scrub had zero hits.

Verified end to end against a live workflow before writing it up:

Function block code Run status What the trace shows
{{ANTHROPIC_API_KEY}} completed, len: 108 {{ANTHROPIC_API_KEY}}
{{ZZ_DOES_NOT_EXIST_XYZ}} completed, len: 25 {{ZZ_DOES_NOT_EXIST_XYZ}}

A resolved secret and a name that never existed render identically, both in a successful run.

Why the claim is hedged

An adversarial pass killed three stronger claims from the first draft, all checked against the sim source:

  1. unredacted: false does not mean "this is a mask." MIN_SUBSTITUTABLE_LITERAL_LENGTH is 8, so a secret shorter than 8 characters is never masked at all and its {{KEY}} is always an unresolved reference. secrets list proves a name exists, never that it resolved in this run. The original line would have told an agent to leave a genuinely broken reference in place.

  2. Singling out --follow implied the default run output is masked. It is not — a plain run and a --follow stream both POST the same executeWorkflow operation, and the execute route contains no redaction call. The section now says every live run response is unmasked and only runs get / logs get return the masked copy.

  3. "Passes through unchanged" is wrong for one of three languages. In a shell block an unresolved placeholder becomes the empty string (shell.ts, resolved ? shellExpansion(...) : ''); JavaScript and Python do leave it literal.

Net effect is a weaker but true claim: a {{KEY}} in a masked log is usually a resolved secret, not proof of one — and the section says what to check when it isn't.

Scope

13 lines added to skills/run-workflow/SKILL.md. Body only — the frontmatter description is unchanged, since this is about behaviour once the skill is routed to, not about routing.

Merged main (the new "four properties of runs and run records" section) and kept both: those bullets are an unheaded continuation of ## Diagnose failures, so they stay attached to the numbered steps and the masking section follows as a peer H2. No contradiction — main's first bullet notes a completed run can sit in redacting status before its content is readable, which is the same pipeline this section describes once it is readable.

bun run check:skills, bun run test, and bun run type-check all pass.

Sim resolves a {{KEY}} reference at execution, then masks the value out
of the log-facing copy that `workflows runs get` and `logs get` return,
writing it back as the {{KEY}} label itself. Agents reading a run read
that as an unresolved reference and "fix" a workflow that already works
- rewriting {{KEY}} into environmentVariables.KEY, hardcoding a literal,
or printing the value to prove resolution.

The skill had nothing on this. It also cannot simply say "it's masked",
because an unknown name is passed through unchanged and renders the same
way, so the check has to key on the name rather than the rendering:
`secrets list` returns names plus an `unredacted` flag.

Also notes that a --follow stream is not a log copy and is not masked.
Both sides added a section to `## Diagnose failures`, so git flagged a
textual conflict where there was no semantic one. Kept both.

main's four run-record properties are an unheaded continuation of
`## Diagnose failures`, so they stay attached to the numbered steps; the
masking section follows as a peer H2, and the closing "Report which mode
ran" paragraph stays last.

No contradiction between the two: main's first bullet notes a completed
run can sit in `redacting` status before its content is readable, which
is the same pipeline the masking section describes once it is readable.
An adversarial review found the first draft wrong in the unsafe
direction on three counts, all verified in the sim source:

1. It said a listed name with `unredacted: false` is a mask, not a
   failure. Unsound: MIN_SUBSTITUTABLE_LITERAL_LENGTH is 8, so a secret
   shorter than 8 characters is never masked at all and its {{KEY}} is
   ALWAYS an unresolved reference. `secrets list` proves a name exists,
   never that it resolved in this run. The old line told an agent to
   leave a genuinely broken reference in place.

2. Singling out `--follow` as "not a log copy" implied the default run
   output IS masked. It is not - a plain run and a --follow stream both
   POST the same executeWorkflow operation, and the execute route
   contains no redaction call at all. Now says every live run response
   is unmasked and only runs get / logs get return the masked copy.

3. "An unknown reference passes through unchanged" is wrong for one of
   three languages: in a shell block an unresolved placeholder becomes
   the empty string (shell.ts `resolved ? shellExpansion(...) : ''`),
   while JavaScript and Python do leave it literal.

Net effect is a weaker but true claim: a {{KEY}} in a masked log is
usually a resolved secret, not proof of one.
@mzxchandra
mzxchandra merged commit d36f82b into main Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant