Skip to content

feat(mcp_readability): reuse unchanged findings so counts track tool changes - #624

Open
akangsha7 wants to merge 1 commit into
mainfrom
evalbench/mcp-readability-carry-forward-wiring
Open

akangsha7 wants to merge 1 commit into
mainfrom
evalbench/mcp-readability-carry-forward-wiring

Conversation

@akangsha7

@akangsha7 akangsha7 commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #623. This is the PR that turns carry-forward on; #622, #615 and #623 are inert without it.

Summary
The judge is a thinking model, so at temperature 0 an unchanged tool surface still yields different findings run to run and issue counts drift with no explanation. With a baseline store configured, an endpoint whose tool surface and judge inputs are unchanged reuses its previous findings verbatim and makes no model call. Numbers then move only for tools a team actually touched, and every departure carries a named reason.

An absent baseline block yields the null store, so a run config that has not opted in behaves exactly as before — including the google3-mirrored one.

Changes

  1. evaluator/mcp_readability/orchestrator.py — builds the store and primes it once, on one thread, before the endpoint pool starts; the per-endpoint alternative is N round trips racing each other. A prime failure sets baseline_unavailable rather than letting every endpoint look like a first run.
  2. Escape hatches — force_refresh, force_refresh_products and EVALBENCH_MCP_FORCE_REFRESH=1, for Guitar one-offs where editing the mirrored config is impractical. Without them a hallucinated P0 could only be cleared by editing the tool.
  3. evaluator/mcp_readability/baseline.py — is_expired ages a baseline out, staggered per endpoint so a fleet primed on one day does not all re-judge on the same later day. The run then reports baseline_expired, so the re-judge is announced rather than an unexplained count change.
  4. scorers/mcp_readability/style_readability.py — run decides before it judges and appends a scope clause in a partial run. The HTML leads with a provenance banner and per-tool markers, so "not re-judged" is distinguishable from "re-judged and scored the same". Provenance is written to its own column only, so the two copies cannot drift.
  5. Second pass — a re-judged tool is described from scratch, so a baseline finding it stops reporting is ambiguous: either the change fixed it or the judge did not mention it. _reconcile_dropped sends just those findings back for a verdict rather than a rewrite, so a still-valid issue is restored with its original wording instead of reading as resolved. A failed reconciliation call leaves the findings in place rather than dropping them.
  6. scorers/mcp_readability/scoring.py — EndpointContext gains baseline, defaulted last so existing positional construction keeps working.
  7. datasets/mcp_readability/run_config.yaml — the baseline block, commented out, documenting the store choices and each deviation reason.
  8. test/mcp_carry_forward_scorer_test.py (new) — drives the real scorer with a recording judge, the only way to assert what matters: zero model calls and byte-identical feedback when nothing changed. test/mcp_readability_test.py adds the orchestrator's side of that, running it twice against a local store, plus the force-refresh hatches; test/mcp_baseline_test.py covers is_expired and its stagger.

@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from e98750e to 9a4e63f Compare September 28, 2026 19:24
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 66dd394 to 5dee51f Compare September 28, 2026 19:25
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 9a4e63f to 96c0e72 Compare September 28, 2026 23:33
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 5dee51f to 175c6e1 Compare September 28, 2026 23:33
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 96c0e72 to 29831c6 Compare September 29, 2026 03:34
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 175c6e1 to 5d9be3c Compare September 29, 2026 03:34
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 29831c6 to efbd5a7 Compare September 29, 2026 03:44
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 5d9be3c to 1598b14 Compare September 29, 2026 03:44
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from efbd5a7 to 79cc805 Compare September 29, 2026 04:32
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 1598b14 to 2ffdf96 Compare September 29, 2026 04:32
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 79cc805 to b78f588 Compare September 29, 2026 04:46
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 2ffdf96 to b00f9b8 Compare September 29, 2026 04:46
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from b78f588 to f9e455e Compare September 29, 2026 05:11
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from b00f9b8 to 530a44e Compare September 29, 2026 05:11
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from f9e455e to a555915 Compare September 29, 2026 19:18
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 530a44e to c55ed58 Compare September 29, 2026 19:19
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from a555915 to 7471754 Compare September 29, 2026 19:52
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch 2 times, most recently from c327667 to 558f3e3 Compare September 29, 2026 20:57
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 7471754 to a504459 Compare September 29, 2026 21:01
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 558f3e3 to 16bc242 Compare September 29, 2026 21:01
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from a504459 to c6dac20 Compare September 29, 2026 22:12
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 16bc242 to 9e514f5 Compare September 29, 2026 22:12
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from c6dac20 to b8d9aa5 Compare September 29, 2026 22:14
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch 2 times, most recently from 70a9160 to cc6151f Compare September 29, 2026 22:18
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from b8d9aa5 to acd3ace Compare September 29, 2026 22:30
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from cc6151f to f6e5ff6 Compare September 29, 2026 22:30
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from acd3ace to 543eed4 Compare September 29, 2026 22:50
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from f6e5ff6 to 9c4a0c7 Compare September 29, 2026 22:50
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 543eed4 to 93251bc Compare September 29, 2026 22:53
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 9c4a0c7 to a258b61 Compare September 29, 2026 22:53
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 93251bc to 6ff5574 Compare September 29, 2026 23:21
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from a258b61 to 7fed539 Compare September 30, 2026 00:26
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 6ff5574 to baabe21 Compare September 30, 2026 00:45
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 7fed539 to 136755c Compare September 30, 2026 00:50
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from baabe21 to 39b447a Compare September 30, 2026 01:06
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch 2 times, most recently from 66f011c to 35933d7 Compare September 30, 2026 01:18
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 39b447a to 0ba1389 Compare September 30, 2026 03:02
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch 2 times, most recently from 99cb7ed to 2a97d80 Compare September 30, 2026 04:58
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 0ba1389 to 34f3752 Compare September 30, 2026 19:59
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 2a97d80 to c116c6b Compare September 30, 2026 19:59
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward branch from 44d8651 to 8409098 Compare October 1, 2026 19:30
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from c116c6b to 29f40d1 Compare October 1, 2026 19:30
Base automatically changed from evalbench/mcp-readability-carry-forward to main October 1, 2026 20:56
@akangsha7 akangsha7 self-assigned this Oct 1, 2026
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from ce03001 to 73941ab Compare October 1, 2026 21:14
…changes

Turns carry-forward on. The orchestrator primes the baseline store once per
run and hands each endpoint its previous judgement; the scorer re-judges only
the tools whose rendered surface changed, carries the rest verbatim, and
reports a named reason for every departure.

An absent baseline block yields the null store, so a run config that has not
opted in behaves exactly as before.
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-carry-forward-wiring branch from 73941ab to 145eb02 Compare October 2, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant