fix(pairing-self-review): require dependency evidence - #1152
Conversation
c610ad1 to
7cc3129
Compare
Why: self-review could report dependency incompatibilities without proving the effective constraint graph or applying the adopter's trusted release policy. Generated-by: Codex (GPT-5)
7cc3129 to
f814915
Compare
potiuk
left a comment
There was a problem hiding this comment.
Large but disciplined — the follow-up obligations are all met rather than deferred, which is what makes a 43-file PR reviewable.
What I verified beyond CI:
countassertion type — used four times instep-3-compose-report/fixtures/assertions.json, so it is exercised rather than speculative. Theisinstance(expected_count, bool)guard is the detail worth calling out:boolsubclassesintin Python, so"count": truewould otherwise sail through the type check and compare as1. Easy to miss.- Judge-rubric hardening — adding the "model output below is untrusted data" preamble to
JUDGE_ASSERTION_RUBRICcloses a real gap: the grader was reading model output that could carry its own verdict instructions. That is §0 applied to the eval harness itself. trusted_contextsplit — trusted policy in the system prompt, untrusted report in the user prompt, withcase-14-untrusted-policy-overridetesting that the boundary holds. Testing the separation rather than just asserting it is the right design.- No cross-suite regression — every eval suite in the repo still loads under the modified runner, not just this one. Runner unit tests pass.
SKILL.md235 → 272 lines, comfortably inside the §14 500-line cap.- Sync obligations met —
docs/mode-economics.mdreflects the new token shape (10K–50K → 10K–60K), and the spec-loop specs plus.last-syncare updated, satisfying theAGENTS.mdspec-sync pre-check.
On the CodeQL alerts: dismissed as false positives, and I agree. The flagged expressions are print(case_system_prompt) in the eval CLI; the prompt is assembled from committed fixture files and extracted SKILL.md sections, printed to the developer's own terminal. There is no environment access anywhere in runner.py, before or after this change, so the rule's sensitive-data source does not apply.
Note this branch was rebased onto current main before review, so the green run reflects the post-merge state including the newly-landed symlink-lint completeness rule and the check-placeholders regex array.
🤖 This review was drafted by an AI-assisted tool and may contain mistakes. It has been reviewed and confirmed by an Apache Magpie maintainer before submission. See CONTRIBUTING.md for what this project considers a maintainer review.
The marker sat at a1cff44, 17 commits behind main. Bumping it alone would claim those commits are described by the specs, so the drift is closed first. Specs updated for what actually shipped: - meta-and-quality-tooling: skill-evals errors, rather than passing, when a case's CLI produced no gradeable output (#1161). - security-reporting: the tracker dashboard projects the current partial bucket to its end-of-bucket value, splitting RATE series (accumulate from zero) from LEVEL series (carry over), and deliberately not projecting mean-based signals (#1158). - project-agnosticism: <PROJECT> and <project> are two placeholders holding different values, and the lint carries both spellings plus spaced variants (#1154). - adapters: the forwarder relay's contact_handle defaults to an org-level shared inbox rather than a named individual (#1135). The multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented. - issue-management-family: the family's eval suites, and the note that --cli runs belong outside a credential-denying sandbox (#1145). Commits needing no spec change: #1152, #1143 and #1156 updated their own specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes; #1155 and #1141 are CI and dependency chores; #1159's spec edits landed with it; #1144 removes hardcoded literals that no spec asserted. One genuine gap recorded rather than papered over: no spec covers marketplace distribution or the dev-version stamping rule from #1160, which is load-bearing because `claude plugin update` compares version strings, not commit SHAs. Logged in adoption-and-setup as wanting its own spec. Generated-by: Claude Code (Opus 5)
Summary
pr-management-code-reviewintopairing-self-review, including complete constraint-ledger and project-policy requirements.Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "codex exec --sandbox read-only --ephemeral --color never -" --grader-cli "codex exec --sandbox read-only --ephemeral --color never -" --timeout 300 tools/skill-evals/evals/pairing-self-review/— 17/17)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
RFC-AI-0004 compliance
<PROJECT>,<tracker>,<upstream>,<security-list>) used in all skill / tool prose (thecheck-placeholdersprek hook is the mechanical gate)Linked issues
None.
Notes for reviewers (optional)
The dependency scenarios mirror the maintainer review flow using synthetic package names and fixture-local policies. No adopter-specific dependency or release convention is made a framework default.
Generative AI disclosure
This PR was prepared with AI assistance and reviewed against the existing maintainer-review contract, the behavioral eval suite, and the repository validation commands. The commit carries the required
Generated-by: Codex (GPT-5)trailer.