[ronniegeraghty]

What changed

Use the pinned native github/evals-action for before/after evals of each changed existing plugin skill, and publish the approved compact comparison comment.

• Hold evals, graders, fixtures and neighboring skill sources at the PR merge revision. Switch only the selected skill between exact base-parent and PR-head SHAs; reject measurement-file drift.
• Run both arms sequentially using Vally 0.21, one saved model, one worker/trial per case and zero retries. A failing Before verdict does not prevent After.
• Show a separate visible case-by-model table for each changed skill, passing/total grader counts, visible After failures, collapsed Before failures, and collapsed marketplace coverage/failure details. Missing/error/skipped evidence never counts as passing.
• Inventory local skills and remote skill folders from exact pinned GitHub trees. Unchanged skills are explicitly not evaluated; loading context is not cross-skill regression coverage. No new model matrix or paid all-marketplace policy is introduced.
• Keep native structured reports; derive only safe case/model/check verdicts from local native JSONL for the comment. Do not upload raw trajectories, model output, grader evidence, exceptions or transcripts.
• Separate approval-gated model permissions from comment-write permissions. Check the current PR head, bot ownership and run/attempt order before updating one comment. Forks are excluded; no pull_request_target or custom Run-button infrastructure.

Approved illustrative sample: #15 (comment) . All sample results, models and comparisons are fictional. The workflow preserves saved model selection; it does not automatically run the sample's two-model/full-cohort example.

Version and changelog

Repository workflow/reporting only. The preserved pending merge integrates already-merged main changes without another plugin version bump.

Checks

41 focused model-free comment/workflow/report checks, skill/eval lint, catalog checks and diff checks pass. The actual pinned catalog inventory resolves 12 skills, six with repo evals. GitHub CI and fresh Copilot review are being requested on the published head. No paid eval or comparison workflow was dispatched; live protected-environment/billing integration is not claimed.

Required setup and risk

A repository owner must configure the existing evals environment with required maintainer reviewers, and the organization must permit Copilot workflow-token billing. Preflight fails closed when protection cannot be confirmed. This PR changes no repository/organization settings, secrets, branch protection, visibility or billing policy.

Maintainers must review workflow/eval/setup/grader code and cost before approving. Trusted same-repository evaluation code receives model-billing credentials. Docker-default configuration and its image/bootstrap/Plugin Run limitations are tracked separately in #17; Vally 0.21 honors that native backend configuration when present. This PR does not claim complete network/credential/host isolation or parity with the locally pinned Copilot binary.

Reporting uses a separate pull-requests: write/actions: read job without model-billing permission. It reads current-run minimal artifacts (seven-day retention subject to repository policy) and links the native run. Plain-language failure reasons describe failed checks, not an AI-generated root-cause diagnosis.

Source contracts

Action pinned to 6e4c077e (v0.2.0): local diagnostics/result outputs, native JSONL evidence, merge revision policy.

No bespoke eval runner, grader framework, regression-only gate, dashboard or scheduled updates.

[Copilot]

🟡 Changes recommended

Missing working-directory inputs prevent both eval invocations from running, and the selection job lacks the actions: read permission needed for private-repository environment checks.

2 open issues

• Add required working directories to both eval action calls · New
• Grant the selection job actions:read for private environments · New

What changed in this PR

Introduces approval-gated before/after evaluations for changed existing plugin skills, preserving the marketplace’s shared evaluation setup while switching only the selected skill’s source.

Changes:

• Adds sequential, pinned evals-action runs against exact base and PR-head skill versions.
• Requires protected-environment approval for same-repository PRs and skips skills without a valid comparison pair.
• Adds model-free workflow checks and documents setup, billing, and execution boundaries.

File	Description
README.md	Introduces the approval-gated PR evaluation workflow.
docs/​checks-and-evals.md	Documents comparison inputs, approval requirements, and execution limits.
.github/​workflows/​skill-evals.yml	Adds skill selection, source switching, and sequential evaluation jobs.
.github/​workflows/​evals.yml	Updates comments to distinguish manual smoke evaluations from skill comparisons.
.github/​scripts/​validation.test.mjs	Adds checks for workflow policy, selection, source restoration, and measurement drift.

_{Review effort: Max (auto)}

[ronniegeraghty]

🧪 Skill eval comparison

Mock-up only: all results are fictional. Example PR changes marketplace-maintainer/release-plugins and pr-description/pr-description.

Changed skills: 📈 2 resolved · 📉 1 new failure. Other skills: 📉 1 new failure. Failure totals count grader/model results.

Sources: Before <base SHA> → After <PR head SHA> · Evidence: <workflow run link>
Trials: One per scenario/model/arm, zero retries. Observed differences do not prove causation.

Read cells as Before → After. A scenario is one test case; graders are its individual checks. Scenario labels below omit their repeated <eval>- prefix; evidence retains the full native IDs.

Changed skill: marketplace-maintainer/release-plugins

Eval : scenario (passing / total graders)	Claude Sonnet 5.5	GPT-6.1 Terra
release-plugins : keeps-already-bumped-version	✅ 3/3 → 3/3	✅ 3/3 → 3/3
release-plugins : retires-approved-plugin	✅ 3/3 → 3/3	✅ 3/3 → 3/3
release-plugins : ignores-local-copy-edit	✅ 4/4 → 4/4	✅ 4/4 → 4/4
release-plugins : asks-for-unknown-release-base	📈 3/4 → 4/4	✅ 4/4 → 4/4
release-plugins : preserves-unreleased-initial-version	📉 4/4 → 3/4	✅ 4/4 → 4/4

Changed skill: pr-description/pr-description

Eval : scenario (passing / total graders)	Claude Sonnet 5.5	GPT-6.1 Terra
pr-description : drafts-structured-pr-body	📈 3/4 → 4/4	✅ 4/4 → 4/4
pr-description : ignores-bug-and-review-question	✅ 4/4 → 4/4	✅ 4/4 → 4/4

❌ After failures in changed skills (1 grader/model result)

Skill: marketplace-maintainer/release-plugins

Eval : scenario: release-plugins : preserves-unreleased-initial-version

Change	Failed grader	Model	Why it failed	Evidence
📉 New in After	skill-invocation (skill routing)	Claude Sonnet 5.5	Required release-plugins activation was not recorded.	<after grader evidence link>

🕰️ Before failures in changed skills: 2 grader/model results, both resolved After

Skill: marketplace-maintainer/release-plugins

Eval : scenario: release-plugins : asks-for-unknown-release-base

Change	Failed grader	Model	Why it failed	Evidence
📈 Resolved in After	program (release-evidence request)	Claude Sonnet 5.5	Mentioned the missing released base but did not ask for it.	<before grader evidence link>

Skill: pr-description/pr-description

Eval : scenario: pr-description : drafts-structured-pr-body

Change	Failed grader	Model	Why it failed	Evidence
📈 Resolved in After	program (PR-body sections)	Claude Sonnet 5.5	Omitted the required Risks / Rollback section.	<before grader evidence link>

🌐 All marketplace skills: 6 evaluated · 📉 1 new failure in an unchanged skill · 6 not evaluated

Cells: passing scenarios / total scenarios, Before → After. A scenario passes only when it meets its configured scoring rule; these examples require every grader to pass.

Skill (passing / total scenarios)	Claude Sonnet 5.5	GPT-6.1 Terra
marketplace-maintainer/release-plugins	🔄 4/5 → 4/5 (1 resolved, 1 new failure)	✅ 5/5 → 5/5
marketplace-maintainer/update-remote-pins	✅ 3/3 → 3/3	✅ 3/3 → 3/3
plugin-authoring/add-plugin	✅ 2/2 → 2/2	✅ 2/2 → 2/2
plugin-authoring/submit-plugin	✅ 2/2 → 2/2	✅ 2/2 → 2/2
plugin-authoring/try-plugin-locally	📉 2/2 → 1/2	✅ 2/2 → 2/2
pr-description/pr-description	📈 1/2 → 2/2	✅ 2/2 → 2/2
eval/compare	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval
eval/create	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval
eval/improve	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval
eval/review	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval
eval/run	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval
eval/setup	⏭️ Not evaluated — no repo eval	⏭️ Not evaluated — no repo eval

The six remote Evals skills have no repository evals. Loading them is not behavioral coverage.

🔎 Marketplace failure details: 2 After · 2 Before (grader/model results)

Skill: marketplace-maintainer/release-plugins

Eval : scenario: release-plugins : asks-for-unknown-release-base

Change	Failed grader	Model	Why it failed	Evidence
Before (resolved After)	program (release-evidence request)	Claude Sonnet 5.5	Mentioned the missing released base but did not ask for it.	<before grader evidence link>

Eval : scenario: release-plugins : preserves-unreleased-initial-version

Change	Failed grader	Model	Why it failed	Evidence
After (new)	skill-invocation (skill routing)	Claude Sonnet 5.5	Required release-plugins activation was not recorded.	<after grader evidence link>

Skill: pr-description/pr-description

Eval : scenario: pr-description : drafts-structured-pr-body

Change	Failed grader	Model	Why it failed	Evidence
Before (resolved After)	program (PR-body sections)	Claude Sonnet 5.5	Omitted the required Risks / Rollback section.	<before grader evidence link>

Skill: plugin-authoring/try-plugin-locally

Eval : scenario: try-plugin-locally : prepares-current-checkout-session

Change	Failed grader	Model	Why it failed	Evidence
After (new)	program (local-command path)	Claude Sonnet 5.5	The command omitted the selected plugin's absolute local path.	<after failed assertion/evidence link>

Legend: ✅ All pass in both arms · 📈 More pass · 📉 Fewer pass · 🔄 Same count, different failures · ❌ Failed grader. Change labels distinguish new, resolved and persistent failures.

⚠️ Error · ⏭️ Not run · ❔ Unknown never count as passing. Use 3/4 → ⚠️ Error, not an invented After count; report partial coverage separately.

Same evals, graders and fixtures across arms; only the PR's changed skill sources differ. Links and SHAs are placeholders, not actual run evidence.

[Copilot]

🟡 Changes recommended

With workflow-token billing enabled, PR-editable workflows can bypass maintainer approval while retaining paid Copilot access, creating a critical authorization gap.

4 open issues

• PR-editable workflow can bypass billing approval · New
• Partial grading results produce misleading transition classifications · New
• Retries reject successful selection artifacts from prior attempts · New
• Stale PR comparison runs are not cancelled · New

2 resolved since last review

• Add required working directories to both eval action calls
• Grant the selection job actions:read for private environments

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

The publishing job can execute PR-controlled code with pull-request write permissions without protected-job approval, creating a privilege-escalation risk.

5 open issues

• PR-editable workflow can bypass billing approval
• Track artifact-producing attempts when retrying jobs · New
• Stale PR comparison runs are not cancelled
• Retries reject successful selection artifacts from prior attempts
• Partial grading results produce misleading transition classifications

3 previously missed issues

In code that hasn't changed since last review

Require explicit passing verdicts before declaring resolution

.github/​scripts/​skill-eval-comment.mjs:215

A missing check in the other arm is treated as a passing verdict: Before with completed=true, routing=false and After with only completed=true produces an improving 1/2 → 1/1 cell and labels routing as resolved. Require an explicit passing verdict for the matching check before declaring a resolution; otherwise mark that comparison unknown or incomplete. Apply the same presence check to the headline totals, transition markers, and marketplace comparison counts.

Run the publisher from a trusted revision

.github/​workflows/​skill-evals.yml:237

This job executes the PR's comment helper with pull-requests: write, even when a new, unpaired skill causes the protected eval job to be skipped. A same-repository integration with source-file write access but no workflow-editing or pull-request-write permission can replace that helper and gain the job's broader token without approval. Run the publisher from a trusted revision rather than the PR checkout, bootstrapping this new helper on the trusted branch before enabling the workflow.

Replace native JSON parse errors with a fixed diagnostic

.github/​scripts/​skill-eval-comment.mjs:119

If native stdout contains a non-JSON line after the first JSON record, JSON.parse can include a fragment of that line in its exception message. The CLI logs that message verbatim at line 339, exposing raw evaluator content even when the Action has already classified the evidence as malformed-native-jsonl. Catch the parsing exception here and replace it with a fixed diagnostic.

_{Review effort: Max (auto)}