Skip to content

feat(darwin): flag a leaderboard where every mutant scores the same - #21

Merged
jjohare merged 2 commits into
mainfrom
dream/security-adversarial-2026-10-03
Oct 6, 2026
Merged

jjohare merged 2 commits into
mainfrom
dream/security-adversarial-2026-10-03

Conversation

@jjohare

@jjohare jjohare commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What

A darwin leaderboard where every scored mutant has the identical score means the scorer is not discriminating (a default value, a fallback answer, or a parse error). The 2026-09-07 run and every pinned @metaharness/darwin@0.10.2 receipt since look like this: all mutants at 0.765, Delta over baseline: +0.000. Those runs still exited 0. This PR makes them fail the darwin evaluator. Decision recorded as ADR-0006.

How

  • One parser. parseLeaderboardRows now reads each row's score. The second row parser (SCORED_ROW_RE) is gone.
  • Inside the existing check. checkDarwinBounds calls checkDarwinScoreUniformity(rows). If two or more scored mutants (non-baseline rows) all share one score, it adds the violation score uniformity: all N mutants score S, equal to baseline — scorer not discriminating. When the mutants agree with each other but not with the baseline, the message reads baseline B instead. Fewer than two scored mutants is not a breach.
  • Same channel as the ADR-0005 bounds. verify-entrypoint darwin prints the violation under darwin bounds VIOLATED and exits 3. scripts/darwin-entrypoint.sh is the configured darwin evaluator and passes that exit code through, so the annexe records the REQUIRED evaluator as FAILED. The nightly script needed no change, because it never parses leaderboards itself.
  • Fixtures. The real 2026-09-07 leaderboard is now the uniformity breach case. A new discriminating fixture (darwin-leaderboard-varied.txt) is the compliant 4 + 5 ceiling. six-in-g2 is rebuilt on top of it, so it isolates the candidate bound.

Consequence

Until the scorer or the mutator changes, darwin nights will exit 3 and carry the uniformity line in their receipt. Before this change they passed on a leaderboard that meant nothing.

Tests

  • darwinBounds.test.ts: the parser reads scores. Uniform-equal-to-baseline and uniform-not-baseline cases are flagged. One differing mutant is clean, a lone mutant is clean, and hand-built rows without scores are ignored.
  • index.test.ts: verify-entrypoint darwin exits 3 on the uniform leaderboard and 0 on the varied one.
  • darwinEntrypoint.test.ts: the real script against the stub darwin covers the same two cases.
  • Locally: npm ci, npm run build, npm run lint, npx vitest run --coverage all pass (10 files, 191 tests).

The PR does not touch docs/dream-cycle/LEDGER.md.

🤖 Generated by Claude Code

Dream Machine and others added 2 commits October 6, 2026 13:44
Wire the score-uniformity check into the darwin bound check (ADR-0006).
parseLeaderboardRows now reads each row's score, replacing the second
parser (SCORED_ROW_RE). checkDarwinBounds reports two or more scored
mutants with one identical score as a violation, so verify-entrypoint
darwin and scripts/darwin-entrypoint.sh exit 3 under "darwin bounds
VIOLATED", the channel ADR-0005 bound breaches already use. Fewer than
two scored mutants is not a breach.

The real 2026-09-07 fixture (all 0.765) is now the uniformity breach
case; a discriminating fixture is the compliant ceiling.

Co-Authored-By: jjohare <github@thedreamlab.uk>
@jjohare
jjohare force-pushed the dream/security-adversarial-2026-10-03 branch from 34f973d to 49d9efd Compare October 6, 2026 13:49
@jjohare jjohare changed the title dream(security-adversarial): arwin leaderboard observed (2026-09-07 run; folded receipts feat(darwin): flag a leaderboard where every mutant scores the same Oct 6, 2026
@jjohare
jjohare marked this pull request as ready for review October 6, 2026 13:49
@jjohare
jjohare merged commit 36ea9c5 into main Oct 6, 2026
7 checks passed
@jjohare
jjohare deleted the dream/security-adversarial-2026-10-03 branch October 6, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants