Skip to content

fix(skill-evals): do not scavenge JSON out of a prose step's output - #1169

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-eval-prose-extraction
Sep 8, 2026
Merged

fix(skill-evals): do not scavenge JSON out of a prose step's output#1169
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-eval-prose-extraction

Conversation

@potiuk

@potiuk potiuk commented Sep 8, 2026

Copy link
Copy Markdown
Member

Summary

Running all 75 eval suites turned up four failures in
security-issue-deduplicate/step-3-merge-body — the whole step, every case.
Reading the model's actual output back, it was correct in all four. The
harness threw it away.

extract_json_from_output's last-resort strategy takes the largest balanced
{...} / [...] block in the CLI's stdout. Ordinary Markdown supplies those
freely: a task-list - [ ] parses as the JSON value [], and a link's
[text] as a malformed one. On a step whose output is prose, that scavenged
fragment replaces the whole body — actual becomes [], raw_output no longer
exists, and every assertion addressing it fails against a correct answer.

What the model actually produced

For case-1-basic-merge, graded as a failure:

## Reporter credited as

- Alice Example
- Bob Researcher

## Security mailing list thread

- Alice Example (initial report): https://lists.apache.org/thread/abc123
- Bob Researcher (redirect-chain vector): https://lists.apache.org/thread/xyz789

Both credits on separate lines, both threads, severity Unknown, the widened
version range, the CWE agreement noted — every assertion satisfied. The reported
failure was field 'raw_output' not present in output.

The fix

A suite whose expected side addresses a wrap key (raw_output, stderr,
exit_code — via expected.json or an assertions.json field) is
declaring that the step emits prose. Skip extraction for those and pass the
body through untouched.

This reuses wrap_is_asserted from #1161 against the mirror-image problem:

#1161 stopped the harness passing what it never graded
this stops it failing what it graded wrong

Both are the same underlying confusion — treating "no JSON here" as a fact about
the model rather than a fact about the step.

Verification

  • All four step-3-merge-body cases pass after the change (0 → 4)
  • Two regression tests, both directions: prose containing a task-list stays
    prose; an ordinary JSON step still goes through the extractor
  • prek run --all-files passes (exit 0)
  • Workspace: pytest, ruff, ruff-format, mypy all green

Version bump

Also bumps the framework to 0.2.0.dev202609081000 and regenerates the
manifests, per marketplace-distribution.md. The marketplace is served from
main and claude plugin update compares version strings, not commit SHAs —
so without a bump, adopters are told they are already up to date and never
receive this fix.

Notes for reviewers

This came out of a full 1196-case run. That run is not finished: 850 cases
returned ERROR from quota exhaustion, not defects — the CLI was healthy when
re-tested. 29 genuine failures remain, clustered in a way that suggests more
shared causes rather than 29 separate bugs (issue-reproducer 7,
issue-fix-workflow 4, audit-finding-fix 3, and two suites failing cases with
matching scope-check / drive-by-reformat names). Those are follow-up work,
not in this PR.

Worth stating plainly: this bug and #1161's were both invisible for as long as
they existed, because each produced a plausible-looking result. The eval harness
needs the same scepticism as the things it grades.

🤖 Generated with Claude Code

https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg

`extract_json_from_output`'s last-resort strategy takes the largest
balanced `{...}` / `[...]` block in the CLI's stdout. Ordinary Markdown
supplies those freely: a task-list `- [ ]` parses as the JSON value `[]`,
and a link's `[text]` as a malformed one. On a step whose output is
prose, that scavenged fragment then replaces the whole body — `actual`
becomes `[]`, `raw_output` no longer exists, and every assertion
addressing it fails against output the model got exactly right.

That is what happened to all four cases of
security-issue-deduplicate/step-3-merge-body. Read back, the model's
merged issue body was correct in each: both credits on separate lines,
both mailing-list threads, severity Unknown, the widened version range,
the CWE agreement noted. The harness threw it away and reported four
failures.

A suite whose expected side addresses a wrap key (`raw_output`,
`stderr`, `exit_code`, via expected.json or an assertions.json `field`)
is declaring that the step emits prose. Skip extraction for those and
pass the body through. This reuses `wrap_is_asserted` from apache#1161 on the
mirror-image problem: that change stopped the harness passing what it
never graded, this one stops it failing what it graded wrong.

All four cases pass after the change. Two tests cover both directions —
prose containing a task-list stays prose, and an ordinary JSON step still
goes through the extractor.

Also bumps the framework version to 0.2.0.dev202609081000 and
regenerates the manifests, so the fix reaches installed copies: the
marketplace is served from `main` and `claude plugin update` compares
version strings, not commit SHAs, so a frozen stamp leaves adopters told
they are already up to date.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg
@potiuk
potiuk merged commit 01ab9f6 into apache:main Sep 8, 2026
41 checks passed
potiuk added a commit that referenced this pull request Sep 8, 2026
…self-correcting (#1174)

* fix(marketplace): re-sync magpie-agent-guard with the dev version stamp

`#1169` stamped the framework version to `0.2.0.dev202609081000` across
every ecosystem manifest. `#1170` landed right after it, adding the
`magpie-agent-guard` plugin — a *new* file, branched before the bump and
therefore carrying the previous stamp `0.2.0.dev202609080121`. New files
do not conflict, so both merged cleanly and `main` was left with a family
plugin one stamp behind the root manifest it inherits from.

`check-family-plugins` has failed on `main` ever since, which is why an
unrelated dependabot PR (#1167) shows a red `prek` job it did not cause.

`python3 tools/dev/check-family-plugins.py --fix` output; no hand edits.

Generated-by: Claude Code (Opus 5)

* fix(prek): regenerate the family plugins instead of reporting them stale

The `check-family-plugins` hook ran check-only, so a contributor whose
manifests drifted got an instruction — "run
`tools/dev/check-family-plugins.py --fix`" — rather than a fixed tree. The
generated files are a pure function of the skills' `family:` frontmatter and
`pyproject.toml`'s version, so there is nothing for them to decide: the fixer
is the only correct response to every failure the check can report.

Run the hook as `--fix`. prek fails a run whose hook modified files, so drift
still stops the commit; it now arrives already corrected, needing only a
re-stage — the same convention `end-of-file-fixer`, `trailing-whitespace`, and
`add-license-headers` already follow here.

`--fix`'s exit code is deliberately left alone: the release-bump procedure
invokes it directly in plain shell blocks, where returning non-zero on a
successful regeneration would read as failure.

Scope note: this closes the hand-fix gap for anyone who runs the hooks, but it
cannot catch the cross-PR case that broke `main` this morning — two PRs green
in isolation, conflict-free on merge, out of sync in the result. Only a check
on `main` (or a merge queue) sees that.

Docs describing the hook as check-only updated to match.

Generated-by: Claude Code (Opus 5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant