From fe162d383288ec5c7044dae2e69d7c415871442f Mon Sep 17 00:00:00 2001 From: RasputinKaiser <178525839+RasputinKaiser@users.noreply.github.com> Date: Mon, 20 Jul 2026 23:44:19 -0400 Subject: [PATCH] audit: recalibrate human scoring rubric to be demanding and graduated (U2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrite skills/plugin-audit/references/scoring-rubric.md so full marks require excellence beyond mere validity, credit is graduated (validity earns the middle, not the max), and leanness is rewarded. Add an explicit "Not applicable → redistribute" rule (drop a genuinely-inapplicable dimension/sub-point and spread its weight proportionally, never a free max) with the no-hooks example. Replace the grade bands with the frozen 5-band scale (92-100 Exceptional / 82-91 Strong / 68-81 Solid / 50-67 Needs work / <50 Poor) plus anti-saturation guidance. Six dimensions and nominal maxes unchanged (15/25/20/20/10/10 = 100). No coefficients hardcoded; gradients described qualitatively so doc and score.py stay in sync. validate.py: PASS 11/11. Co-Authored-By: Claude Opus 4.8 --- .../plugin-audit/references/scoring-rubric.md | 85 +++++++++++-------- 1 file changed, 51 insertions(+), 34 deletions(-) diff --git a/skills/plugin-audit/references/scoring-rubric.md b/skills/plugin-audit/references/scoring-rubric.md index 4d99708..532381c 100644 --- a/skills/plugin-audit/references/scoring-rubric.md +++ b/skills/plugin-audit/references/scoring-rubric.md @@ -1,36 +1,48 @@ # Plugin health rubric (100 points) -Score each dimension independently. Deduct only with concrete evidence. Half points allowed. +Score each dimension independently on **graduated** criteria — not pass/fail. Validity earns the MIDDLE of a sub-point's range; full marks require excellence beyond mere validity. Award partial credit that scales with how far a plugin exceeds the bar (leaner, more complete, more robust = more). Deduct only with concrete evidence. Half points allowed throughout. -The deterministic scorer (`score.py`) computes the objective slice of several dimensions and returns `{dimension:{auto,max,needs_judgment}}`; treat `auto` as the machine FLOOR and judge only what it flags. `references/deterministic-scoring.md` maps each dimension to what is auto-scored vs judged. Two runtime diagnostics supply evidence lines: `tokens.py` (token/budget report) feeds Context economy, and `errscan.py` (session-log error scan) feeds Hooks health — cited inline below. +The deterministic scorer (`score.py`) computes the objective slice of several dimensions and returns `{dimension:{auto,max,needs_judgment}}`; treat `auto` as the machine FLOOR (mechanics that are simply present/valid) and judge the quality gradient on top. `references/deterministic-scoring.md` maps each dimension to what is auto-scored vs judged. Two runtime diagnostics supply evidence lines: `tokens.py` (token/budget report) feeds Context economy, and `errscan.py` (session-log error scan) feeds Hooks health — cited inline below. + +## Not applicable → redistribute + +A dimension (or a sub-point) that **genuinely does not apply** is DROPPED, and its weight is redistributed proportionally across the applicable dimensions, renormalizing the total back to 100. It never earns a free max. + +Example: a plugin that correctly ships no hooks and needs none has no Hooks health dimension — drop its 10 points and spread them across the other five in proportion to their nominal maxes, then score out of 100. Likewise, parity sub-points that assume dual-harness support do not apply to a correct single-harness plugin: drop those sub-points and redistribute within the dimension. "Not applicable" means the capability is correctly absent, NOT that it is present but weak — a broken or missing-but-needed component is scored low, not dropped. ## 1. Manifest integrity — 15 points -- 4 — Each present `plugin.json` is valid JSON with kebab-case `name`, semver `version`, accurate `description`. -- 4 — Cross-harness parity: a plugin claiming dual-harness support ships BOTH `.claude-plugin/plugin.json` and `.codex-plugin/plugin.json`, with identical `name` and agreeing `version` across every manifest (ignore any Codex `+build` suffix) — a missing manifest or version drift is a deduction. A single-harness plugin earns full credit for its one valid manifest. -- 3 — All component pointers (`skills`, `hooks`, `mcpServers`, `apps`) present where components exist, `./`-prefixed, inside plugin root. -- 2 — Correct layout: each manifest dir holds only its `plugin.json`/`marketplace.json`; components at root. -- 2 — Publisher metadata appropriate to distribution level: `author` always; interface `displayName`/`shortDescription` if shared; icons/legal links only if published. +Full marks require complete, accurate publisher metadata AND lean, correct pointers — not merely valid JSON. Score each sub-point on the gradient from "present/parses" (middle) to "exemplary" (max). + +- 4 — Manifest correctness. Middle: each present `plugin.json` is valid JSON with kebab-case `name` and semver `version`. Toward full: `description` is accurate, specific, and current (not stale or templated); nothing vestigial. Malformed, drifted, or misleading fields deduct. +- 4 — Cross-harness parity. A dual-harness plugin ships BOTH `.claude-plugin/plugin.json` and `.codex-plugin/plugin.json` with identical `name` and agreeing `version` (ignore any Codex `+build` suffix); a missing manifest or version drift is a deduction. A single-harness plugin: this sub-point does not apply — drop and redistribute within the dimension (its one valid manifest is scored under sub-point 1, not handed a free 4 here). +- 3 — Pointer hygiene. Middle: component pointers (`skills`, `hooks`, `mcpServers`, `apps`) present where components exist, `./`-prefixed, inside plugin root. Toward full: pointers are lean and exact — no dangling, redundant, or unused entries; every declared path resolves. +- 2 — Layout discipline. Each manifest dir holds only its `plugin.json`/`marketplace.json`; components at root. Full credit requires a clean tree with no stray or misplaced files; partial otherwise. +- 2 — Publisher metadata appropriate to distribution level. Middle: `author` present. Toward full: interface `displayName`/`shortDescription` present and polished if shared, icons/legal links present if published. Thin or absent metadata at the plugin's actual distribution level deducts. ## 2. Skill quality — 25 points -Score across all skills, weight by how central each skill is. +Score across all skills, weight by how central each skill is. Full marks require skills that an agent can execute flawlessly without guessing — not just prose that reads plausibly. Credit is graduated per criterion: uneven quality across skills pulls the score toward the middle. -- 8 — Each skill does one job. No kitchen-sink skills; no two skills doing the same job. -- 7 — Bodies are imperative instructions for the agent (verb-first, concrete steps, explicit inputs/outputs) — not user documentation, not marketing. -- 5 — Steps are actionable: file paths, commands, exact formats. An agent could follow them without guessing. -- 5 — Failure handling: what to do when a file is missing, a command fails, or input is ambiguous. +- 8 — Single responsibility. Middle: skills mostly do one job. Toward full: every skill has one crisp job, no kitchen-sink skills, and no two skills overlapping. Any duplication or a bloated catch-all skill deducts in proportion to how central the offender is. +- 7 — Imperative agent instructions. Middle: bodies are mostly directive. Toward full: consistently verb-first, concrete steps with explicit inputs/outputs, zero user-doc or marketing tone. Narrative or explanatory drift deducts. +- 5 — Actionability. Middle: steps are followable in the common case. Toward full: exact file paths, commands, and formats throughout, leaving nothing to guess. Vague or hand-wavy steps deduct. +- 5 — Failure handling. Middle: the main failure mode is addressed. Toward full: missing files, failing commands, and ambiguous input are each handled explicitly across the skills. Silence on failure paths deducts. ## 3. Trigger precision — 20 points -- 7 — Every description states what the skill does AND when to use it, with concrete trigger phrases users actually say. -- 5 — Negative scope: says when NOT to use it, or is precise enough that misfires are unlikely. -- 5 — No trigger collisions: no two skills in the plugin (or obvious user-level skills) claim the same prompt. -- 3 — Risky or niche skills are guarded against accidental firing. On Codex, via `agents/openai.yaml` `policy.allow_implicit_invocation: false`. On Claude Code (which has no opt-out flag), via a tightly negative-scoped description or by shipping the capability as an explicit command. Award full credit to a single-harness plugin that uses its harness's available mechanism. +Full marks require descriptions precise enough that the right skill fires and the wrong one never does — not merely a description that exists. + +- 7 — What + when. Middle: descriptions state what the skill does. Toward full: also state WHEN to use it with concrete trigger phrases users actually say. Generic or when-less descriptions deduct. +- 5 — Negative scope. Middle: scope is implicitly narrow. Toward full: explicit "not for X" clauses, or descriptions precise enough that misfires are genuinely unlikely. Broad, ambiguous scope deducts. +- 5 — No collisions. Middle: no obvious overlap. Toward full: verified no two skills in the plugin (or obvious user-level skills) claim the same prompt space. Any real collision deducts. +- 3 — Guarding risky/niche skills. Middle: risk is acknowledged. Toward full: accidental firing is actively prevented — on Codex via `agents/openai.yaml` `policy.allow_implicit_invocation: false`; on Claude Code (no opt-out flag) via a tightly negative-scoped description or by shipping the capability as an explicit command. A single-harness plugin is scored on its harness's available mechanism; if the plugin has no risky/niche skills, this sub-point does not apply — drop and redistribute within the dimension. ## 4. Context economy — 20 points -Budgets (flag, then deduct): +Reward leanness: credit scales with how far UNDER budget a plugin sits and how genuine its progressive disclosure is — merely being within budget earns the middle, not the max. + +Budgets (flag, then grade the gradient): | Item | Budget | Hard flag | |---|---|---| @@ -38,33 +50,38 @@ Budgets (flag, then deduct): | SKILL.md body | ≤ 600 words | > 1,500 words | | References loaded eagerly | 0 (load on demand) | body inlines reference content | -- 8 — Bodies within budget; detail pushed to `references/` (progressive disclosure). -- 6 — No duplicated content across skills; shared material lives in one reference, pointed to. -- 4 — Descriptions within budget (they load into every session's metadata). -- 2 — No dead weight: unused directories, stale examples, empty files. +- 8 — Body economy + progressive disclosure. Middle: bodies within budget. Toward full: comfortably UNDER budget with detail genuinely pushed to `references/` and loaded on demand — not padded to the limit. Bodies near or over budget, or that inline reference content, deduct. +- 6 — No duplication. Middle: no glaring repeats. Toward full: shared material lives in exactly one reference that others point to; zero copy-paste across skills. Duplicated passages deduct. +- 4 — Description economy. Middle: descriptions within budget. Toward full: tight and well under the char limit while still stating what + when. Bloated descriptions (they load into every session's metadata) deduct. +- 2 — No dead weight. Full credit requires no unused directories, stale examples, or empty files; any dead weight deducts. -Evidence: read the token/budget report (`tokens.py`) here. Its session-tax headline and per-skill budget headroom quantify bloat — a skill over its body or description budget, or a plugin whose session tax has grown against baseline, is a concrete deduction line. A green machine floor from `score.py` (chars/words within budget, no dead files) fixes the mechanical part; judge duplication and progressive-disclosure quality on top. +Evidence: read the token/budget report (`tokens.py`) here. Its session-tax headline and per-skill budget headroom quantify leanness — a skill merely at budget scores mid, one comfortably under scores high, and one over its body/description budget (or a plugin whose session tax has grown against baseline) is a concrete deduction line. A green machine floor from `score.py` (chars/words within budget, no dead files) fixes the mechanical part; judge duplication and progressive-disclosure quality on top. ## 5. Hooks health — 10 points -If the plugin has no hooks and needs none, award 10. +If the plugin correctly ships no hooks and needs none, this dimension does NOT apply — drop it and redistribute its 10 points across the other five dimensions (see the redistribution rule above). Do not award a free 10. Score the sub-points below only when hooks exist; full marks require hooks that are correct at RUNTIME, not just valid on paper. -- 3 — Valid `hooks.json` shape: event → matcher group → handlers. -- 3 — Contracts correct per event (see plugin-hooks skill references): on Codex, Stop returns JSON on stdout, blocking uses documented shapes or exit 2 + stderr. **Static shape is necessary but not sufficient** — a hook whose command errors at runtime fails this criterion even with valid JSON. -- 2 — Paths use `${CLAUDE_PLUGIN_ROOT}` (Codex also accepts `${PLUGIN_ROOT}`); scripts exist and are executable; sensible `timeout`. -- 2 — Hooks respect each target harness's limits rather than silently depending on unsupported behavior: on Codex tool events are Bash-only today and matchers are ignored on UserPromptSubmit/Stop; on Claude Code matchers match all tools. +- 3 — Shape. Middle: valid `hooks.json` shape (event → matcher group → handlers). Toward full: clean, minimal, no vestigial entries. +- 3 — Runtime-correct contracts. Static shape is necessary but not sufficient — full credit requires the hook actually runs without error and honors its event's contract (on Codex: Stop returns JSON on stdout; blocking uses documented shapes or exit 2 + stderr). A hook whose command errors, times out, or misbehaves at runtime deducts even with valid JSON. +- 2 — Paths + executables. Middle: paths use `${CLAUDE_PLUGIN_ROOT}` (Codex also accepts `${PLUGIN_ROOT}`). Toward full: every referenced script exists, is executable, and carries a sensible `timeout`. Absolute/bare paths or missing scripts deduct. +- 2 — Per-harness limits respected. Full credit requires hooks that work within each target harness's actual limits rather than silently depending on unsupported behavior (on Codex tool events are Bash-only today and matchers are ignored on UserPromptSubmit/Stop; on Claude Code matchers match all tools). Latent dependence on unsupported behavior deducts. Evidence: read the runtime error scan (`errscan.py`) here. Hook/tool/skill errors it aggregates for the target plugin are direct evidence against the contracts and paths sub-points — a hook that throws, times out, or hits a missing script at runtime is a real deduction the static shape check cannot see. `score.py` auto-scores only the shape/paths-present part; runtime health and per-harness correctness are judgment on top. ## 6. Distribution readiness — 10 points -- 3 — README covers what it does, the skills it ships, and install steps (per targeted harness). -- 3 — Version discipline: version bumped with changes; changelog or ledger exists if the plugin has history. -- 4 — A marketplace entry exists for each harness the plugin targets, using that harness's own schema: Claude Code `.claude-plugin/marketplace.json` lists the plugin under `plugins[]` with a resolvable `source` (string or object) and a `category` (`policy.*` do not exist here); the Codex registration (`~/.agents/plugins/marketplace.json` or `~/.codex/config.toml [marketplaces]`) carries `policy.installation`, `policy.authentication`, `category`, and a resolvable `source.path`. A targeted harness with no entry is a deduction; do not deduct a Claude Code entry for lacking Codex-only `policy` fields. +Full marks require a plugin someone else could actually find, install, and trust — not just a README that exists. + +- 3 — README. Middle: covers what the plugin does. Toward full: also lists the skills it ships and correct install steps per targeted harness, and is current with the code. Thin, stale, or install-less READMEs deduct. +- 3 — Version discipline. Middle: a version exists. Toward full: version is bumped with changes and a changelog or ledger records history where the plugin has any. Stale versions or missing history deduct. +- 4 — Marketplace entries. A marketplace entry exists for each harness the plugin targets, in that harness's own schema: Claude Code `.claude-plugin/marketplace.json` lists the plugin under `plugins[]` with a resolvable `source` and a `category` (`policy.*` do not exist here); the Codex registration (`~/.agents/plugins/marketplace.json` or `~/.codex/config.toml [marketplaces]`) carries `policy.installation`, `policy.authentication`, `category`, and a resolvable `source.path`. A targeted harness with no entry deducts; a harness the plugin does not target does not apply (drop that share and redistribute within the dimension). Do not deduct a Claude Code entry for lacking Codex-only `policy` fields. ## Grade bands -- 90–100 — Excellent. Maintain; audit after major changes. -- 75–89 — Good. Quick wins available. -- 55–74 — Fair. Prioritized improvement pass recommended. -- < 55 — Poor. Structural work needed before feature work. +Score each dimension on the graduated criteria above so a typical good-but-improvable plugin lands **Solid/Strong (~78–84)**, and reserve **Exceptional** for a plugin that is near-perfect across ALL applicable dimensions. + +- **92–100 — Exceptional.** Strong across every applicable dimension; nothing material left to improve. Rare but achievable. +- **82–91 — Strong.** Solid throughout with a few real wins available. +- **68–81 — Solid.** Mature and working; a prioritized improvement pass pays off. +- **50–67 — Needs work.** Functional but with clear gaps; improve before feature work. +- **< 50 — Poor.** Structural problems; rebuild the weak dimensions first.