fix(darwin): route the ruvllm mutator through a Loom shim; fail uniformity only on zero real mutations

[jjohare]

Why every darwin night scored 0.765

No model scores darwin's mutants. Under --sandbox mock, darwin@0.10.2 scores each variant deterministically from its own surface files; the model is only the mutator. Every mutant scored 0.765 because every mutant was byte-identical to the baseline:

1. Scaffold, not model. RuvllmMutator sends no loom_options. The Loom answered all 20 calls in about 15 ms with an ontology page (_Served verbatim from the Ontology Loom … no model generation was performed._), and darwin's validator rejected each one: mutation rejected by validator; surface unchanged (secret handling). That explains the 1.5 s nightly duration.
2. Hard-coded 30 s abort. With scaffold off, the model takes about 60 s per call. darwin aborts at a fixed 30 s (timeoutMs ?? 30_000) and never reads RUVLLM_TIMEOUT_MS, so the 180000 in dream.config.json was dead config and each call became a silent no-op.

ADR-0006 assumed the scorer was returning a default; ADR-0007 corrects that.

Part 1: the Loom shim (ADR-0007 §2)

When the command uses --mutator ruvllm, scripts/darwin-entrypoint.sh starts a loopback shim (packages/cli/src/loomShim.ts) and points darwin at it. The shim:

• forces loom_options.scaffold=false and a max_tokens floor of 8192;
• sends headers at once (darwin's abort covers only the header wait) and enforces RUVLLM_TIMEOUT_MS upstream, capped at 280 s;
• replaces scaffold, truncated, empty and error replies with {"choices":[]}, which darwin records as a no-op, and logs one line per call plus a per-verdict summary into the receipt.

The script also refuses --mutator ruvllm without --ruvllm-url, and the --flag=value spellings. darwin reads only --flag value, so --sandbox=mock used to pass the guard while darwin ran the real sandbox.

Part 2: amended uniformity rule (owner decision, ADR-0007 §3b)

ADR-0006's check exists to catch an inert pipeline, which is what hid the scaffold defect for a month. A run where the model really edited the surfaces but nothing moved the mock score is an honest "no score-moving edit" result, not a failure.

The shim rewrites its counts to a stats file after every call. The entrypoint passes that file as verify-entrypoint --shim-stats, and checkDarwinBounds(rows, 0, { realMutations }) applies:

Mutator	Shim ok	Uniform leaderboard	Varied
ruvllm	> 0	exit 0, note no improvement found: N real mutations, all scored S (= baseline)	exit 0
ruvllm	0	exit 3: 0 real mutations: the mutator made no edit	exit 0
ruvllm	summary missing or unreadable	counted as 0, so exit 3 (a broken shim can't pass silently)	exit 0
deterministic (no shim)	—	exit 3, ADR-0006 unchanged (nothing to count)	exit 0

Evidence

• Real run (darwin@0.10.2 from npm, real Loom, about 18 min): loom-shim: summary ok=20 scaffold=0 truncated=0 empty=0 upstream-error=0. The leaderboard is still uniform at 0.765: the only score-relevant edits (slice(0,30)→40, maxAttempts 3→4) fall between mock-ladder rungs, which need width 50 for 0.875 and 70 for 0.985 (checked with darwin's own scorer).
• That real leaderboard under the amended rule, with --shim-stats ok=20: exit 0, darwin: no improvement found: 20 real mutations, all scored 0.765 (= baseline). With the summary missing: exit 3.
• Control: the deterministic mutator through the same entrypoint gives a varied leaderboard (winner g2_v5 0.875, delta +0.110) and exits 0.

Tests

Tests were written failing first. vitest run: 223/223 pass; eslint is clean.

• loomShim.test.ts: request rewrite, response classification, early headers, timeout.
• darwinEntrypoint.test.ts: the real script with a stub darwin and a fake Loom, covering scaffold off, a slow answer outliving the header abort, refusals, and wiring cases (a) ok=0 + uniform → 3, (b) ok>0 + uniform → 0 with the note, (c) no summary + uniform → 3, (d) varied → 0.
• index.test.ts and darwinBounds.test.ts: the same table at the CLI and pure-function levels, plus an unreadable summary and the generation bounds still enforced alongside a pass.

Not changed / follow-ups

• dream.config.json is unchanged; nightly runs keep --mutator ruvllm.
• Upstream asks for @metaharness/darwin (ADR-0007 §3c): forward loom_options, read RUVLLM_TIMEOUT_MS, and pass stdout traces to the mutator.
• Verified from agentbox against the Loom door; not yet run on HP's annexe.

🤖 Generated by Claude Code