You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix marketplace eval false negatives and first-release routing - #13
Follow up on the three case failures recorded during #12 without claiming that the shared environment caused them. Fix two deterministic grader false negatives and clarify the release skill's routing for confirmed first-release notes when no file changes are needed.
Vally runs program checks in a disposable copy and changes trajectory.workDir. Use the source-bound runtime activation path to identify the original selected plugin directory instead of comparing its correct absolute command against the grading copy. Preserve wrong-source, no-launch, no-install, no-mutation and unverified-status checks; accept explicit approval of tool use as permission wording.
Recognize an actionable release-evidence request expressed as a numbered or bulleted list while still rejecting bare missing-base mentions and unsupported readiness claims.
Keep eval prompts, fixtures, target-activation requirements, negative cases and scoring thresholds unchanged. The before/after Actions workflow is separate and is not included here.
Version and changelog
marketplace-maintainer: 0.1.0 -> 0.2.0, with matching manifest/catalog and changelog entry. Ronnie confirmed the last pre-change commit, ab32b6a39b174d7ab8e6b3bf731984d555baf642, as the released 0.1.0 comparison base and requested the next 0.2.0 public preview. The plugin README describes first-release note support. No other plugin source or version changed.
Checks
50 retained model-free checks passed, including a real native gradeInDisposableWorkspace regression and wrong-path/no-launch countercases. Skill/eval lint, local catalog matching and diff checks passed.
With explicit approval, ran both affected evals once through Vally 0.19.0: 7/7 real cases passed (release-plugins 5/5, try-plugin-locally 2/2), observed model claude-haiku-5.5, one worker, one trial per case, zero retries. Run ID: fe1a12ed-27b1-4fef-be07-e971557ff6fd.
The first-release case now records source-bound release-plugins activation. Both negative cases still record no forbidden activation. All seven original runtime system-message inventories advertise all 12 marketplace skills.
Original failed results remain unchanged. Deterministic diagnostics confirm the candidate accepts the two original valid answers even in a relocated grading copy; these diagnostics do not relabel or overwrite historical grades.
This is not a controlled causal comparison or a full 16-case rerun. The earlier run auto-selected Copilot 1.0.106-8; this run disabled auto-update and used the repository-pinned 1.0.98. No claim is made that the description change alone caused improved routing.
Raw logs, reports and preserved workspaces are retained locally and excluded from this PR. No CI workflow was dispatched.
Risk
The plugin change is instructions-only: a more explicit routing description, with no new tools, hooks, permissions, dependencies or network destinations. Grader code executes only when the affected evals are run; it reads recorded trajectory data and synthetic fixture files, without installing plugins or starting nested Copilot sessions. Existing native invocation and side-effect boundaries remain enforced.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The list-header pattern can falsely accept a plugin-name-only request by matching an unrelated missing-base mention in an earlier sentence on the same line.
This PR addresses marketplace evaluation false negatives and clarifies release-skill routing for confirmed first releases that require no file changes.
Changes:
Use runtime skill activation paths to identify the original plugin directory during disposable grading, and recognize explicit tool-use approval.
Recognize release-evidence requests in numbered or bulleted lists, with regression coverage for both graders.
Clarify first-release routing and documentation, and bump marketplace-maintainer to 0.2.0.
Reject a preceding missing-base statement followed by an unrelated list request; cover period and semicolon forms while preserving actionable multiline evidence requests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The whole-list match combines unrelated requests and evidence mentions: both Current blockers:\n- The released base is missing.\n- Please provide a plugin name. and Please provide:\n- A plugin name. The released base is missing. pass without requesting release evidence. Require request wording in the header before inheriting it into list items, and limit inherited requests to each item's first sentence/clause. Add negative regressions for both forms.
Require request wording in the list header and evidence in each item's first clause instead of combining unrelated requests and mentions across the whole list. Cover both fresh Copilot examples and preserve explicit and inherited valid requests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The grader rejects otherwise valid responses with bold Markdown request headers, requiring a matching fix and a regression test to prevent formatting-only failures.
A Markdown header such as **Please provide:** followed by - A released base. is rejected, although the identical list without bold formatting passes. The closing ** prevents this pattern from matching, and the statement-based fallback cannot associate the request with its list item. Allow closing emphasis after the colon and add a bold-header regression so formatting alone does not fail the eval.
Allow closing emphasis after the request colon while retaining per-item evidence boundaries; cover a bold valid request and a bold unrelated-item negative.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Follow up on the three case failures recorded during #12 without claiming that the shared environment caused them. Fix two deterministic grader false negatives and clarify the release skill's routing for confirmed first-release notes when no file changes are needed.
trajectory.workDir. Use the source-bound runtime activation path to identify the original selected plugin directory instead of comparing its correct absolute command against the grading copy. Preserve wrong-source, no-launch, no-install, no-mutation and unverified-status checks; accept explicit approval of tool use as permission wording.Version and changelog
marketplace-maintainer: 0.1.0 -> 0.2.0, with matching manifest/catalog and changelog entry. Ronnie confirmed the last pre-change commit,ab32b6a39b174d7ab8e6b3bf731984d555baf642, as the released 0.1.0 comparison base and requested the next 0.2.0 public preview. The plugin README describes first-release note support. No other plugin source or version changed.Checks
gradeInDisposableWorkspaceregression and wrong-path/no-launch countercases. Skill/eval lint, local catalog matching and diff checks passed.release-plugins5/5,try-plugin-locally2/2), observed modelclaude-haiku-5.5, one worker, one trial per case, zero retries. Run ID:fe1a12ed-27b1-4fef-be07-e971557ff6fd.release-pluginsactivation. Both negative cases still record no forbidden activation. All seven original runtime system-message inventories advertise all 12 marketplace skills.Risk
The plugin change is instructions-only: a more explicit routing description, with no new tools, hooks, permissions, dependencies or network destinations. Grader code executes only when the affected evals are run; it reads recorded trajectory data and synthetic fixture files, without installing plugins or starting nested Copilot sessions. Existing native invocation and side-effect boundaries remain enforced.