eval: Responses API, repeat-based variance, and harness tests

[NullLabTests]

Why

The eval harness reported a single number per scenario with no indication of spread. With LLM_API_KEY unset it runs mock plans that always pass, so npm run eval in CI proved nothing about real model behaviour and nothing about whether the scoring code still worked. The LLM path also hard-coded /chat/completions with a gpt-4o default, had no retry or timeout, and no tests at all.

Changes

• eval/llm.ts — Responses API by default, auto-detecting /chat/completions for non-OpenAI gateways (vLLM, Ollama, Together, Groq). Strict JSON Schema on the plan instead of json_object, so responses parse without coercion. Retry with exponential backoff plus jitter on 408/409/425/429/5xx, per-request timeout, and coerceJson() for fenced or prose-wrapped output.
• eval/metrics.ts — aggregateRepeats(), mean(), stdev(), and summarizeAggregates() for mean ± sd per scenario.
• eval/runner.ts — EVAL_REPEATS (default 3), per-scenario aggregates, and a stable flag that is false unless every repeat was valid.
• eval/metrics.test.ts — 8 tests covering the stats helpers, JSON coercion, and gateway URL handling.
• CI — adds npm test and a mock-mode npm run eval job. Adds permissions: contents: read and concurrency cancellation.

Verification

npm run validate   0 errors, 0 warnings, 5 informational
npm run coverage   77 atom files
npm run lint       93 files, no fixes applied
npm run check      tsc --noEmit clean
npm test           8 passed
npm run eval       6 scenarios, all valid (mock mode)

Note on reading the numbers

EVAL_REPEATS defaults to 3 because a single sample cannot distinguish a reliable result from a lucky draw. Aggregate output marks a scenario stable only when all repeats were valid. estimateTokens() is still a 4-chars-per-token heuristic, so relative comparisons hold but absolute token counts should not be quoted as measurements.

Not run: live LLM mode, which needs LLM_API_KEY. The API-shape changes are compile-checked and unit-tested against a stub, not exercised against a real endpoint.

🤖 Generated with Claude Code