Conversation
akangsha7
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
September 22, 2026 01:19
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 28, 2026 19:24
e98750e to
9a4e63f
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 28, 2026 19:25
66dd394 to
5dee51f
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 28, 2026 23:33
9a4e63f to
96c0e72
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 28, 2026 23:33
5dee51f to
175c6e1
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 03:34
96c0e72 to
29831c6
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 03:34
175c6e1 to
5d9be3c
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 03:44
29831c6 to
efbd5a7
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 03:44
5d9be3c to
1598b14
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 04:32
efbd5a7 to
79cc805
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 04:32
1598b14 to
2ffdf96
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 04:46
79cc805 to
b78f588
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 04:46
2ffdf96 to
b00f9b8
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 05:11
b78f588 to
f9e455e
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 05:11
b00f9b8 to
530a44e
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 19:18
f9e455e to
a555915
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 19:19
530a44e to
c55ed58
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 19:52
a555915 to
7471754
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
2 times, most recently
from
September 29, 2026 20:57
c327667 to
558f3e3
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 21:01
7471754 to
a504459
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 21:01
558f3e3 to
16bc242
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 22:12
a504459 to
c6dac20
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 22:12
16bc242 to
9e514f5
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 22:14
c6dac20 to
b8d9aa5
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
2 times, most recently
from
September 29, 2026 22:18
70a9160 to
cc6151f
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 22:30
b8d9aa5 to
acd3ace
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 22:30
cc6151f to
f6e5ff6
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 22:50
acd3ace to
543eed4
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 22:50
f6e5ff6 to
9c4a0c7
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 22:53
543eed4 to
93251bc
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 29, 2026 22:53
9c4a0c7 to
a258b61
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 29, 2026 23:21
93251bc to
6ff5574
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 30, 2026 00:26
a258b61 to
7fed539
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 30, 2026 00:45
6ff5574 to
baabe21
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 30, 2026 00:50
7fed539 to
136755c
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 30, 2026 01:06
baabe21 to
39b447a
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
2 times, most recently
from
September 30, 2026 01:18
66f011c to
35933d7
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 30, 2026 03:02
39b447a to
0ba1389
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
2 times, most recently
from
September 30, 2026 04:58
99cb7ed to
2a97d80
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
September 30, 2026 19:59
0ba1389 to
34f3752
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
September 30, 2026 19:59
2a97d80 to
c116c6b
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward
branch
from
October 1, 2026 19:30
44d8651 to
8409098
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
October 1, 2026 19:30
c116c6b to
29f40d1
Compare
Base automatically changed from
evalbench/mcp-readability-carry-forward
to
main
October 1, 2026 20:56
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
October 1, 2026 21:14
ce03001 to
73941ab
Compare
…changes Turns carry-forward on. The orchestrator primes the baseline store once per run and hands each endpoint its previous judgement; the scorer re-judges only the tools whose rendered surface changed, carries the rest verbatim, and reports a named reason for every departure. An absent baseline block yields the null store, so a run config that has not opted in behaves exactly as before.
akangsha7
force-pushed
the
evalbench/mcp-readability-carry-forward-wiring
branch
from
October 2, 2026 00:45
73941ab to
145eb02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #623. This is the PR that turns carry-forward on; #622, #615 and #623 are inert without it.
Summary
The judge is a thinking model, so at temperature 0 an unchanged tool surface still yields different findings run to run and issue counts drift with no explanation. With a baseline store configured, an endpoint whose tool surface and judge inputs are unchanged reuses its previous findings verbatim and makes no model call. Numbers then move only for tools a team actually touched, and every departure carries a named reason.
An absent baseline block yields the null store, so a run config that has not opted in behaves exactly as before — including the google3-mirrored one.
Changes