Skip to content

audit: recalibrate human scoring rubric — demanding & graduated (U2) - #13

Merged
RasputinKaiser merged 1 commit into
mainfrom
worktree-agent-a36c160fefd4a3e4a
Jul 21, 2026
Merged

audit: recalibrate human scoring rubric — demanding & graduated (U2)#13
RasputinKaiser merged 1 commit into
mainfrom
worktree-agent-a36c160fefd4a3e4a

Conversation

@RasputinKaiser

Copy link
Copy Markdown
Owner

Recalibrates the plugin-audit human scoring rubric (skills/plugin-audit/references/scoring-rubric.md) so it discriminates across the quality range instead of saturating. Part of the audit-recalibration batch (Unit U2).

What changed

  • Graduated, demanding criteria for all six dimensions: validity now earns the MIDDLE of each sub-point's range; full marks require excellence beyond mere validity (leaner bodies well under budget, complete/accurate publisher metadata, runtime-correct hooks, precise triggers). Credit described as a qualitative gradient — no coefficients hardcoded, so the doc stays in sync with score.py.
  • "Not applicable → redistribute" rule added near the top: a genuinely-inapplicable dimension or sub-point is dropped and its weight spread proportionally, renormalizing to 100 — never a free max. Includes the no-hooks example; single-harness parity and no-risky-skills sub-points now drop-and-redistribute within their dimension.
  • Frozen 5-band scale replaces the old bands: 92–100 Exceptional · 82–91 Strong · 68–81 Solid · 50–67 Needs work · <50 Poor, with one line of anti-saturation guidance (typical good-but-improvable plugin lands Solid/Strong; Exceptional reserved for near-perfect across all applicable dimensions).

Invariants held

  • Six dimensions and nominal maxes unchanged: Manifest 15 · Skill 25 · Trigger 20 · Context 20 · Hooks 10 · Distribution 10 = 100. Sub-point maxes still sum to each dimension.
  • Dual-harness-neutral wording preserved. No versions/CHANGELOG/README/manifests/scripts/other skills touched — only scoring-rubric.md.

Verification

  • python3 scripts/validate.pyPASS 11/11, exit 0.
  • Code review (agent): dimension sums = 100, redistribution rule internally consistent, frozen bands verbatim, all relative links resolve. No findings.

🤖 Generated with Claude Code

… (U2)

Rewrite skills/plugin-audit/references/scoring-rubric.md so full marks require
excellence beyond mere validity, credit is graduated (validity earns the middle,
not the max), and leanness is rewarded. Add an explicit "Not applicable →
redistribute" rule (drop a genuinely-inapplicable dimension/sub-point and spread
its weight proportionally, never a free max) with the no-hooks example. Replace
the grade bands with the frozen 5-band scale (92-100 Exceptional / 82-91 Strong /
68-81 Solid / 50-67 Needs work / <50 Poor) plus anti-saturation guidance.

Six dimensions and nominal maxes unchanged (15/25/20/20/10/10 = 100). No
coefficients hardcoded; gradients described qualitatively so doc and score.py
stay in sync. validate.py: PASS 11/11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@RasputinKaiser
RasputinKaiser merged commit 7ed35e8 into main Jul 21, 2026
2 checks passed
@RasputinKaiser
RasputinKaiser deleted the worktree-agent-a36c160fefd4a3e4a branch July 21, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant