Recalibrate audit scorer: de-saturate & discriminate (U1) - #14
Merged
Conversation
Replace the bottom-compressed, saturated auto-scorer with a graduated,
N/A-redistributing engine so scores discriminate across the quality range.
- N/A redistribution: each dimension carries `applicable` + `ceiling`; a
dimension that doesn't apply (no hooks, no skills) is DROPPED from both the
numerator and denominator of the effective total. No more free 10 for a
no-hooks plugin.
- score_skill_quality: flat 0 -> graduated per-skill proxies (imperative body
~8, actionable specifics ~5, failure handling ~5; ~7 judgment residue). A good
plugin now earns ~16; a poor one ~0.
- score_context: budget-utilization gradient (full at <=50% of budget, linear
decay to 0 at >=100%) for bodies (600w) and descriptions (400c) — a 599w body
scores far below a 300w one.
- score_trigger: NOT-clause credit now requires a redirect to an alternative
("not for X, use Y"), not a bare "not" token.
- score_manifest: coarse 4/2/0 steps softened into finer gradients.
- score_hooks: graduated when present, N/A (redistributed) when absent.
- Effective total renormalized to /100 with a `grade` field (bands:
92+ Exceptional / 82 Strong / 68 Solid / 50 Needs work / <50 Poor).
- Schema freeze respected: total.{auto,max}, dimensions[k].{auto,max,
needs_judgment} preserved; only additive keys. --min-baseline now skips
pre-recalibration baselines (scale change) instead of false-failing.
- Selftest: 4 band fixtures (poor/fair/solid/excellent) + de-saturation
invariants (hooks-absent dropped, skill_quality>0, lean>near-budget context,
92+ achievable). deterministic-scoring.md rewritten to match exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RasputinKaiser
added a commit
that referenced
this pull request
Jul 21, 2026
Recalibrated both audit scorers to be demanding and discriminating: - Graduated credit + leanness rewards; skill_quality gains machine signal (floor 60->74.4) - N/A dimensions redistribute weight instead of a free max (hooks no-hooks free-10 removed) - Frozen 5-band grade scale (Exceptional/Strong/Solid/Needs work/Poor); 92+ genuinely hard - NEW scripts/calibrate.py saturation meter + CI guardrail; self-score floor 57->70 - Integration fix: repoint calibrate selftest fixtures to score.py tier builders Landed via PRs #10-#14. Merged-tree: validate 11/11, curator 87/87, score+calibrate selftests green, calibrate --check passes over the real inventory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rewrites
scripts/score.py(the plugin-audit scorer engine) andskills/plugin-audit/references/deterministic-scoring.mdto fix a bottom-compressed, saturated auto-score. Measured over 14 real plugins the auto-score ranged 28–60 (mean 43),skill_qualitywas0/25for every plugin, andhooks_healthwas 93% saturated (no hooks → free 10).Changes
applicable+ceiling. A dimension that doesn't apply (no hooks, no skills) is dropped from both sides of the effective total — no free maximum.skill_quality: flat0→ graduated per-skill proxies (imperative body ~8, actionable specifics ~5, failure handling ~5; ~7 judgment residue). Good plugin ~16, poor ~0.context_economy: budget-utilization gradient — full at ≤50% of budget, linear decay to 0 at ≥100% — for bodies (600w) and descriptions (400c). Rewards leanness.trigger_precision: NOT-clause credit now requires a redirect to an alternative ("not for X, use Y"), not a barenottoken.manifest_integrity: coarse 4/2/0 steps softened into finer gradients (validity, pointers, layout proportional).hooks_health: graduated when present, N/A (redistributed) when absent.gradefield. Bands: 92+ Exceptional · 82 Strong · 68 Solid · 50 Needs work · <50 Poor.Schema freeze
total.{auto,max}anddimensions[k].{auto,max,needs_judgment}preserved; only additive keys (applicable,ceiling,grade,total.effective). Nominal maxes stay 15/25/20/20/10/10. CLI/exit contract unchanged.--min-baselinenow skips pre-recalibration baselines (scale change) rather than false-failing. Verifiedportfolio.py's consumption path still works.Verification
python3 scripts/score.py selftest— 4 band fixtures (poor/fair/solid/excellent) + de-saturation invariants (hooks-absent dropped,skill_quality> 0, lean > near-budget context, 92+ achievable, monotonic ordering).python3 scripts/validate.py→ PASS 11/11 ·curator.py selftest→ 83/83 · schematotal.max==100.skill_quality14 (was 0), hooks dropped (not free 10).pathlikeover-match tightened, baseline scale guard added).Note: CI
--minfloor is intentionally left for the coordinator to reset post-merge (the effective-scale total differs from the old raw sum).🤖 Generated with Claude Code