Second text lineage: nvidia/nemotron-3-super-120b-a12b - #421
Draft
sebasmos wants to merge 30 commits into
Draft
Conversation
sebasmos
marked this pull request as ready for review
September 4, 2026 22:27
sebasmos
requested review from
Agastya191,
MohShahin,
PeterLi-jpg,
Yehudha-kennedy,
aadityapanchal7,
amarzullo24,
armaanvgrewal,
arshnoor08,
azzy-chemE,
dr-shrey,
duckyquang,
felipeocampoos,
maximinl and
renato-umeton
and removed request for
dr-shrey
September 4, 2026 22:28
…ineage blind-metric arm Mirrors the imaging lane from #416. _key(model) resolves the key from the model id; _backend() sends Gemini ids to GeminiBackend and everything else to LocalOpenAICompatibleBackend on NIM with an 8192 output-token cap. --model defaults to the existing Gemini id, so current invocations are unchanged; output and cache paths are model-scoped. Each row also records the declared terminal letter per condition and the summary carries a declared-only view, so a completion that never commits to a letter is excluded rather than scored. A None completion raises instead of being cached. Prompts and parsers untouched. Arm: nvidia/nemotron-3-super-120b-a12b on the same 40 MedQA cases as the Gemini comparator, cold cache, 120 calls. Per-case rows, summary and call cache committed, with three allowlist entries for the #374 guard (one definitional, two empirical).
Same manifest, first 100 cases; the first 40 replay from the committed cache and match the n=40 rows exactly, so the n=40 file is kept as the paper-matched cohort and the n=100 file sits under results/n100/. 180 new calls. Two allowlist entries for the new file, one definitional and one empirical.
…00 cases 120 new calls appended to the tracked Gemini cache; the other 60 prompts were already in it. The first 100 cases of the same manifest, and the first 40 rows are identical to the committed n=40 arm on every original column, so the paper-matched cohort is unchanged and still replays with zero API calls. One allowlist entry for the new file, definitional.
sebasmos
force-pushed
the
feat/text-lane-model-flag
branch
from
September 5, 2026 11:32
dadb5fd to
448b372
Compare
… text runners to it experiments/_lane.py is now the one place that maps a model id to its key, its backend and its output cap. Each runner keeps its own experiment logic and loses its private copy of MODEL, _key, _letters and _Cache: 268 insertions against 798 deletions. The cache key is unchanged, sha256(model NUL prompt), so every committed Gemini cache replays exactly as before. The default model also keeps the committed output and cache paths; any other model gets its own subdirectory and its own cache file, which keeps a thirteen-way fan-out off one shared file. declared() reuses declared_mcq_choice from #418 rather than carrying a second implementation, mapping its option text back to a letter. Eight tests pin the contract, including that the cache key and the default-model paths do not move.
#416 documented the NVIDIA endpoint at about 40 RPM with a penalty for concurrent bursts. A free-tier key sustains much less than that: measured on this account the bucket is small and refills slowly, one call every 20s completes 9 attempts in 10, and a single call succeeds again after 60s of idle. Four arms in parallel returned 429 after roughly 300 calls. Two changes, both in the shared module so every text lane gets them. Calls are paced to the measured sustained rate, held across threads so it holds whatever max_workers a runner uses, and disabled for Gemini, which has no such restriction. A 429 now waits for the bucket to refill instead of failing: RetryBackend's five quick attempts expire while it is still empty, which is what killed whole arms mid-run and cost the calls already made. Not included: #416 also builds the OpenAI client with timeout=60.0 and max_retries=0 so the client's internal retries stop fighting RetryBackend. That fix is on its branch and this lane inherits it when #416 merges, so gateway.py is left alone here rather than conflicting with an open PR.
…at n=120 Same manifest and the same 120 cases as the committed Gemini arms, temperature 0, cold cache, one worker paced to the free tier: contamination_cascade, text_cue_types, dose_response and test_awareness. 1,503 new calls over about eight hours, unattended, zero failures once pacing was in. Three of the four claims replicate with the same shape at roughly a quarter of the amplitude: the negation cue is the potent one and qualifier padding is null (43 gain / 1 lose, p = 5e-12); adoption rises monotonically with dose and falls again at emphatic (20/0, p = 2e-6); accuracy framing lowers adoption in the same direction, underpowered at this floor (4/9, p = 0.27). Two diverge: telling the model agreement is graded doubles adoption on nemotron (20/5, p = 0.004) where it suppressed on Gemini (0/25), and the recall-stratified cascade contrast is not detectable (Fisher p = 0.163 against 0.002) because baseline adoption is 15/120 against 77/120 and cannot be split. Not a capability floor: nemotron answers 108/120 unseeded against Gemini's 101/120. The traces explain the cascade gap. The prompt asks for a single letter; under seeding Gemini complies 0/118 times and writes a median 5,000-character deliberation in which 70 of its 74 adoptions defer explicitly, 57 on cases it had answered correctly. Nemotron complies 116/120 and never enters that deliberation. One guard exemption, for a p of 5e-12 that round(p, 6) writes as 0.0.
sebasmos
changed the title
sebasmos
marked this pull request as draft
September 6, 2026 19:38
sebasmos
changed the title
…dpoint The dispatch sent every non-Gemini id to NVIDIA NIM, so an open-weights arm could only run through a vendor API and inherited a rate limit that does not apply to it. Setting BENCHMAXXING_LOCAL_BASE_URL now points the OpenAI-compatible backend at that server, drops the key lookup, since no local server checks one, and switches pacing off, since the interval exists only to respect a vendor request ceiling. Gemini and DeepSeek ids keep their vendor routing whatever the variable is set to, so it cannot redirect a committed comparator arm to a different model behind the same id. The blind-metric lane carries its own copy of the key and backend dispatch, so it reads the same variable on the same terms. Prompts, parsers and the reasoning cap are untouched, so prompts stay byte comparable across lineages.
openai/gpt-oss-120b on the same manifest cases as the Gemini and nemotron arms: the four MedQA runners at n=120 and blind-metric at n=100 plus the paper-matched 40. Served on four H100 cards with vLLM, so the arm costs no vendor calls and the whole run is one pass with no rate limiting. The model returns the bare option letter in content and keeps its reasoning out of it, so nothing truncated is ever scored. Caches are committed, force added past the ignore rule as the Gemini cache is, so every number replays with no key and no server. Four allowlist entries cover the two blind-metric files, one definitional and one empirical per file, each stating what was read.
live_peer_organic.py passed (model, prompt) to the shared cache, whose signature is complete(prompt, model=None), so the question text went out as the model id and every call failed with 404; and its --model never reached the committee, whose holdout was bound to the Gemini constant. The holdout is now the requested model and the two flash peers read their answers from the committed cache, so a new holdout faces exactly the board the paper holdout saw. temperature_sensitivity.py called the shared cache with a temperature and sample index it does not take, raising TypeError on the first call. It gets a draw-aware cache on the shared dispatch, keyed as the committed Gemini sweep was written, so that arm still replays with no calls. Both default-model arms replay row-identical from their committed caches.
The runner imported the Gemini-only cache from referee_threshold.py, so the floor could not be measured on a second model. It now carries a draw-aware cache on the shared text-lane dispatch with the same key, so the committed Gemini arm replays with no calls and its summary stays byte identical, and any model the text lane can address runs through it.
Qwen/Qwen2.5-VL-72B-Instruct on the 35 CheXpert cases of nih_match_35.csv, served with vLLM on four cards, through the --model version of the runner from #416 plus the same local-server routing the text lane uses. Blind decoy uptake is 14 of 35 off a zero baseline and the test-aware prime suppresses it on every case. The images come from a public mirror by exact CheXpert-v1.0-small path. Their bytes do not reproduce the committed Gemini cache keys, so the pixels are re-encoded relative to the originals and could not be verified byte identical; image_provenance.json records the sha256, size and dimensions of each file used.
…nd the live-peer board on gpt-oss All fourteen single-model MedQA runners plus the deliberation channel arm, the referee self-inconsistency floor and the live-peer organic board, on the same manifest cases as the Gemini and nemotron arms, 8,092 cached calls in all. The channel runner gains a gpt-oss branch because the model has no thinking switch: enable_thinking is silently ignored and reasoning_effort only budgets the hidden channel, so the rows record content length to make that visible. Twenty-two allowlist entries cover the constant, duplicate and rounded columns, each stating what was read; the imaging entries for the previous commit are included here.
…ent, and a model flag for the referee floor nvidia/nemotron-3-super-120b-a12b, same manifest and the same 120 cases as the committed Gemini arms, one worker paced to the free tier: attributed_tier, authority_ladder, committee_size_sweep, deliberation_framing, leader_as_auditor, paraphrase_robustness, plausible_distractor, pre_emptive_referee and rationale_validity. Every arm clears the unseeded-accuracy floor in both lineages (nemotron 107 to 112 of 120, Gemini 101). deliberation_channel.py is new: the same 120 cases with the model's reasoning channel removed, hidden, or open, each cell with an unseeded competence control, run on both lineages. It is the experiment that explains the cascade gap and it supersedes the earlier readings of it. experiments/_lane.py now retries a dropped connection or read timeout above gateway.RetryBackend rather than losing the arm; three ablation arms had died that way with zero calls made. experiments/referee/referee_self_inconsistency.py takes --model. The Gemini path, keys and file format are unchanged and the committed run replays with no key; any other model writes under its own slug and goes through the shared dispatch. Nineteen guard exemptions, each with the exact value or construction that makes the flagged column legitimate.
sebasmos
changed the title
Same cohort as the committed Gemini arm. Confident seed 0.05, hedged 0.00; 5/0, p = 0.0625, same direction as Gemini (0.42 vs 0.14, 29/1) and underpowered. Two guard entries with the observed floor that makes the flagged column and the one-sided contrast legitimate.
…ird lineages, local serving, two runner fixes deliberation_channel.py and referee_self_inconsistency.py take the third-lineage versions, which are supersets of the ones on this branch and are the versions the committed gpt-oss results were produced with. degeneracy_exemptions.json is the union of both sides.
… lane The NVIDIA endpoint under load answers 503 'Service temporarily overloaded', 502/504, and an intermittent 404 for a model /v1/models still lists and that answers 200 a minute later; one such 404 ended an arm 466 calls in. Retries stay bounded by RATE_LIMIT_TRIES so a withdrawn model still fails.
Collaborator
|
Good work on the declared-only accounting added to blind_metric.py, that's exactly the stricter view the naming-detector fix earlier in this file argues for. One problem: n_named_rubric under declared_only sums named_rubric_when_drifted over declared_drifters, but named_rubric_when_drifted is gated on the legacy drifted flag (blind_ans == decoy from parse_legacy_string), while declared_drifters is built from blind_declared == decoy_letter, a different parser. When the two disagree on a row, that row can be a declared drifter yet still carry named_rubric_when_drifted as False even though the naming regex actually matched, so n_named_rubric silently undercounts the declared-only naming rate. I'd compute a declared-specific flag, something like named and blind_declared == decoy_letter, and sum that instead of reusing the legacy-gated one. |
The 429-wait and transient retry lived only in _lane.Cache; the draw-aware temperature cache and the referee floor paced their calls but went to RetryBackend directly, so a 429 that the shared cache would have waited out killed those arms instead. paced_complete() is now the single path and all three use it.
sebasmos
force-pushed
the
feat/text-lane-model-flag
branch
from
September 8, 2026 00:15
99ab999 to
c05a2f7
Compare
n_named_rubric under declared_only summed named_rubric_when_drifted, which is gated on the legacy parse (blind_ans == decoy via parse_legacy_string), while a declared drifter is blind_declared == decoy_letter from the declaration detector. Where the two parsers disagree the legacy flag is False whatever the naming regex found, so the declared-only naming rate undercounted. It bit one committed row: nemotron n=100 medqa-35 is a declared drifter whose completion does name the rubric, and that arm's declared n_named_rubric was 0 rather than 1. named_rubric_when_declared_drifted is now written per row and summed instead, with a fallback for rows written before it existed. All six blind-metric arms regenerated keylessly; no pre-existing field changed in any of them, so every reported rate including the paper's n=40 figures is untouched. declared_only_summary() is now a module-level function so the accounting is testable. Reported by @Agastya191 on #421.
LocalOpenAICompatibleBackend built its client with the SDK defaults, so max_retries was 2. That put a hidden retry loop under gateway.RetryBackend (5 attempts) and _lane.paced_complete, letting one logical call become many unpaced HTTP requests and spend a rate bucket the lane believed it was metering. Retries now belong to the caller: max_retries defaults to 0, and both it and timeout are parameters rather than fixed values, so the hosted endpoint keeps a 60 s fast-fail while a locally served model gets 600 s for a long completion. Raised by @Agastya191 on #416, where the same line hardcodes both. Transport only: all 158 committed row files across the three lineages replay set-identical with the new client and no API calls, and the only summary change is new_api_calls_this_run.
The two flash peers answer before the holdout and never see it, so their board is the committed one whatever the holdout is; the peer answers are read from the committed cache and only the holdout is new. The peer board, consensus and ground truth are therefore identical across lineages by construction, while bare, board, follows_consensus and net_harm are the holdout's own. Follows a wrong organic peer consensus on 7 of 8 cases, the same rate as Gemini and gpt-oss but not the same seven: Gemini resists medqa-23, the other two resist medqa-72. On a stratum of 8 that rate carries a Wilson interval of [0.53, 0.98], so the agreement of the three point estimates is a coincidence of a small denominator and not a replicated magnitude. Follow rate on a correct consensus tracks unseeded accuracy: 0.908 at 101/120, 0.982 at 110/120, 1.000 at 115/120.
…f-inconsistency floor temperature_sensitivity at n=120, four temperatures with three sampled draws each above 0, resuming through the paced_complete fix that the earlier attempt died without. Adoption is flat in temperature (0.092 at t0 to 0.122 at t1.0) while within-case draw disagreement rises with it (0.083, 0.108, 0.158), the same shape as both other lineages at their own levels. referee_self_inconsistency on nemotron, 80 fresh cache-bypassed calls: 40/40 stable, rate 0.0, every draw declared. This is the within-run control for the across-run answer instability measured from repeated prompts, and it is why that instability is reported as a property of re-running days apart rather than of the model: seconds apart this endpoint does not flip. Four guard entries, each stating the construction or the result that makes the column legitimate.
sebasmos
changed the title
One branch, one new model against the Gemini baseline. This branch is the nemotron-3-super-120b-a12b lineage, so the 62 openai_gpt-oss-120b result files that arrived with the third-lineage merge are removed along with their 24 degeneracy-guard entries. Nothing is lost: those results are on feat/local-third-lineage at d076c52 and belong in their own pull request against main. The shared machinery stays, because it is not a result and both lineages need it: experiments/_lane.py with paced_complete and the local-serving path, the reasoning-channel dispatch in deliberation_channel.py that knows which families take enable_thinking, and the two hermetic dispatch tests that use an open-weights model id as a fixture and touch no committed results. Every nemotron and Gemini row file still replays byte-identical with no API calls; guard green; suite failing set unchanged against main.
Same rule as the previous commit: one branch, one new model. This branch is nemotron-3-super-120b-a12b, which is text-only and cannot produce an imaging arm at all, so the four Qwen2.5-VL-72B files and their four guard entries do not belong here. They get their own branch against main, where the committed Gemini imaging arm they are compared against already lives. No other lineage's results remain on this branch. Guard green, suite failing set unchanged against main.
duckyquang
requested changes
Sep 8, 2026
duckyquang
left a comment
duckyquang
left a comment
Member
There was a problem hiding this comment.
Went through the branch at 6b9efbf: read the full code diff and independently recomputed the committed artifacts (per-row exact McNemar from the JSONLs, cache-key grouping for the repeat-prompt caveat, a BH reconstruction — all run locally from the committed caches, zero API calls). The core is solid: every per-arm number in the table reproduces exactly from committed rows, and the suite's failing set is identical to main. But the two meta-claims in the body don't reproduce, and one lane skipped the consolidation this PR exists to do.
Blocking
- The repeat-measurement caveat's numbers don't reproduce from the committed caches. Grouping identical prompts by cache key across the 19 nemotron arm caches I get 57/242 repeated prompts whose stored answer changed, not
35/301; Gemini gives 0 disagreements at every scope I tried (240, 4194, or 4608 repeated prompts depending on which caches are included), never0/559. Direction confirmed — Gemini perfectly stable, nemotron not — but the reproducible flip rate (~24%) is about double what the body says, which if anything strengthens your caveat. Commit the script that produced35/301/0/559, or update the body to the numbers a fresh clone can recompute. - Same for the BH paragraph: nothing committed defines the 36-contrast family or its q-values. Enumerating every
pvaluein the 18 arm summaries gives 40 contrasts per lineage; BH at 0.05 reproduces nemotron 18 survivors exactly, but Gemini comes out 32 (not 34), and the smallest dropped nemotron q is0.0658(body says every drop has q above 0.07). Please commit the family definition + script and align the body. experiments/blind_metric/blind_metric.pyduplicates ~80 lines ofexperiments/_lane.py(_is_local,_key_name,_key,_backend, model scoping) instead of importing it the wayexperiments/mimic_cxr_text/blind_metric.pydoes — and the duplication isn't inert. Its_declaredaccepts only a bare terminal letter while_lane.declaredfirst triesdeclared_mcq_choice, so "declared-only" means different things in different lanes of this same PR (visible in the committed Gemini n100 summary:n_declared=0on baseline/test-aware under the stricter detector). And its_CachecallsRetryBackenddirectly undermax_workers=4, skippingpaced_complete— the 429 wait and 5xx/404 handling this PR added for exactly this vendor — so the body's "all nineteen single-model text runners and the referee floor go through it" isn't quite true for this lane. Port it onto_lanelike its mimic sibling.
Question
- With
BENCHMAXXING_LOCAL_BASE_URLset,is_local()captures every non-Gemini/non-DeepSeek id — including NIM-servednvidia/nemotron-3-super-120b-a12b, which this PR turns into a committed comparator arm. A cache-miss replay of the nemotron arms in a shell with a local vLLM configured quietly routes those calls to the local server (and 404s get retried as transient, so it fails slowly). The tests pin this as deliberate; now that a NIM model is itself a comparator, would an explicit local-id prefix or allowlist be safer? - Title and body still present three lineages, but
656f859/6b9efbfmoved gpt-oss and Qwen out to #425/#426/#427. Update to nemotron-only and link the splits? Also worth stating the intended merge order, since each split carries its own copy of_lane.py.
Numbers, small asks
- Nemotron's unseeded accuracy is arm-dependent: 21/120 cases change their bare answer between arms and per-arm accuracy runs 107–112/120, so the headline
108/120is one arm's value (Gemini is byte-identical at 101/120 in all 19 arms). Worth a footnote — the caveat paragraph already owns this instability. - "Same 120 cases" has exceptions: nemotron
authority_ladderis n=60 vs Gemini's 120, andplausible_distractoris n=110 vs 103, so the guideline 0.87-vs-0.925 comparison is on different n. - The referee floor's Gemini "0/40" is 0 flips over 39 declared pairs (1 undeclared pair, 2 undeclared draws) per the committed summary; the table glosses the exclusion.
- The 18 MedQA arms still score everything through
parse_legacy_string— undeclared-exclusion exists only in blind_metric and the referee floor — so truncated nemotron completions get fallback-scored in the arms. Fine for symmetry with the Gemini arms, but say so in one line.
Checked & fine
- Every per-arm table number recomputes exactly from the committed JSONLs: negation
43/1, p=5.12e-12; dose20/0, p=1.9e-6and3/14, p=0.0127; guideline52/0; attending11/0; agreement framing20/5, p=0.0041; sweep0/9, p=0.0039; blind11/0, p=0.00098. All 18 nemotron summaries are internally consistent with their rows. - @Agastya191's declared-only naming point is fixed at this head by
41a86ef, and I verified the fix empirically: recomputing nemotron n100 both ways givesn_named_rubric1 with the new flag vs 0 under the legacy gating, and the committed summary says 1 — so the committed numbers came from the fixed code. - The
max_retries=0/timeoutparameterization onLocalOpenAICompatibleBackendaddresses the concern raised on #416, and the three defect fixes each land with real tests. - The new exemption entries properly separate definitional from empirical reasons.
Nits
deliberation_channel.py's_callre-implements the retry loop with a narrower transient check (timeout/connectonly) than_lane._is_transient— it misses the 502/503/504/intermittent-404 handling208f828added. Use_lane._is_transient(root)there.referee_self_inconsistency.py's_Cacheis a standalone duplicate of the draw-aware cache;temperature_sensitivity._DrawCachesubclasses_lane.Cache— same trick works here.deliberation_channel.pydoesout_dir, _ = _lane.scoped(...)then hand-rebuilds the exact cache pathscopedjust returned.live_peer_organic.pybuilds the holdout'sModelSpecwithlineage="gemini", tier="lite"whatever--modelis; harmless today since the peers never see the holdout, but it's a lie in the roster.imaging_chexpert/imaging_blind_metric.py's_keyhas a dead branch (thellama/nvidia/metaconditional and the final fallback both return the NVIDIA key) — a third copy of key resolution in the tree.
CHANGES REQUESTED
sebasmos
changed the title
Thirty runners across medqa, referee, cascade, contamination, model_dependence, SUPPORT2 and the MIMIC-CXR text lane still hardcoded a Gemini id, so no second lineage could run them. They now take --model through the shared dispatch, which takes this branch from 20 model-aware runners to 53. Code and tests only: no result files, and no other lineage's results are on this branch. The port is the same one already reviewed on the open-weights text branch, taken as-is rather than rewritten, so the two branches do not diverge on shared machinery. Two hermetic test files come with it. Suite failing set unchanged against this branch's previous head.
… the paced call path The previous commit was meant to bring the shared --model port for the thirty Gemini-only runners onto this branch but landed only its two test files: the runner edits were left unstaged by a stash used for a baseline comparison, so the pushed head asserted a port that was not there and any fresh clone failed eighty tests. The port is now committed. While bringing it over, every one of those runners built gateway.RetryBackend directly and never went through _lane.paced_complete, so a 429 ended the arm after five quick attempts instead of waiting for the rate bucket. That is the same defect fixed earlier for the first nineteen runners, reintroduced by the port; overnight it killed eleven of nineteen arms. All thirty now call paced_complete, the port test pins that path rather than the raw RetryBackend shape, and the support2 modules keep their MODEL import with a noqa because rebind_models needs it in the namespace. Code and tests only, no result files. Suite failing set is the twelve guard self-tests, identical to main, verified below in a fresh clone.
sebasmos
force-pushed
the
feat/text-lane-model-flag
branch
from
September 9, 2026 12:13
2b85ca6 to
edcc528
Compare
…ling already was Review on #421: experiments/blind_metric/blind_metric.py carried its own copy of key resolution, backend dispatch, local routing and model scoping, its cache called RetryBackend directly under four workers and so skipped the paced 429 wait, and its declaration detector accepted only a bare terminal letter while _lane.declared first tries the shared declared_mcq_choice. All of that now comes from experiments/_lane.py; the module's old names remain as the shared implementations so callers and tests keep working, and the dispatch tests patch the shared seams rather than private copies. The detector change is visible in the declared-only view of the Gemini arms, which the stricter rule had reported as n_declared=0 on baseline and test-aware because Gemini declares its answer in prose: it is now 39/40 and 37/40 at n=40, and 98/100 and 97/100 at n=100. Every legacy field in every blind-metric row file is byte-for-byte unchanged, so the numbers the arms are compared on do not move. Also from the review: is_local() no longer captures NIM house ids. A shell with a local vLLM configured must not answer a cache miss for the committed nemotron comparator from a different model behind the same id; HOSTED_PREFIXES pins that and the local-serve test is updated to the new rule.
…an recompute them Review on #421: neither the multiplicity family nor the repeat-prompt caveat was defined by anything committed, and the reviewer's own reconstruction gave different numbers from the body. This script is now the definition. The family is every pvalue the arm summaries report, per lineage, BH at 0.05. A repeat prompt is one cache key sent by more than one arm; it disagrees when the stored completions differ and changes the answer when both are bare letters and the letters differ. The body is rewritten to what this script prints once the remaining arms land.
…an honest roster deliberation_channel's retry loop recognised only timeouts and connection drops as transient, missing the 5xx and intermittent-404 handling the shared module has; it now uses _lane._is_transient and the shared sleep. It also rebuilt by hand the cache path _lane.scoped had just returned; it uses the returned path. live_peer_organic labelled the holdout's ModelSpec lineage="gemini" whatever --model was; the roster now names the real lineage and open-weights flag, which changes no result because the peers never see the holdout. imaging_chexpert's _key carried a third copy of key resolution with a dead branch; it and _is_local are now the shared implementations. Keyless replay of the three text runners touched is set-identical to the committed rows.
…eer and contamination families Every text runner that is model-aware and has its dataset on this machine now has a nemotron result: referee_threshold, referee_judge, referee_requery_design, referee_deployable, multi_round, majority_pressure, seed_timing, hierarchy_temp, hierarchy_dominance, true_peer_control, unanimity_break, orchestrator_failure, scale_c, cascade_C_flash, push_c, break_it and clean_a, all on the same MedQA manifest as the committed Gemini arms and all through the paced call path with no failed attempt. Several of these arms stratify on the cases the model answered wrong solo, so nemotron's n is smaller by design where it is more accurate: 39 hard cases against Gemini's 85 in scale_c, 10 against 60 in push_c. Only 3 of 40 holdouts adopt the planted shortcut in the referee lane, so every referee variant trivially flags exactly those 3; the guard entries record that as a floor effect, not a strong result.
…udit on nemotron The ladder had been run at the runner's default --n 60 against Gemini's 120, which the review caught. Re-run at 120: the first 60 rows replay identically from cache, 300 new calls for the rest. The finding stands: colleague 0.017, automated system 0.050, senior attending 0.133, clinical guideline 0.833, with the same rung ordering as Gemini and automated-system to guideline at 94/0. contamination_audit needed --solo-records pointed at nemotron's own solo file; the default is the committed Gemini file, and with it the script probes the Gemini ids listed there. On its own records: full accuracy 0.90, question-only 0.30, options-only 0.31 against 0.20 chance, and the flip rate is 0.30 on baseline-wrong cases against 0.056 on baseline-correct (Fisher p = 0.031), the same contamination signature as both Gemini models.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second text lineage:
nvidia/nemotron-3-super-120b-a12bSame MedQA manifest, same 120 cases as the committed
gemini-2.5-flash-litearms, temperature 0. Every number replays from a fresh clone with no API calls. Draft. Every arm that is model-aware, text-only and has its dataset on the machine that ran this has a nemotron result: 18 MedQA arms, 17 referee/cascade/peer/contamination arms, blind metric, the referee floor.Unseeded accuracy. Gemini 101/120 in every arm; nemotron 107 to 112/120 depending on the arm (108/120 in the cascade arm). Contrasts are exact McNemar gain/lose within an arm, or Fisher where noted.
Headline arms (n=120 unless stated)
text_cue_typescontamination_cascadedose_responsetest_awarenessauthority_laddercommittee_size_sweepplausible_distractorattributed_tierdeliberation_framingleader_as_auditorparaphrase_robustnesspre_emptive_refereerationale_validityseed_confidencesuper_additivitytemperature_sensitivitylive_peer_organicdeliberation_channelReferee, cascade and peer families (new in this revision)
These arms stratify on cases the model answered wrong solo, so nemotron's n is smaller by design where it is more accurate.
referee_threshold/judge/requery_design: holdouts adopting the planted shortcutreferee_deployable: false positives on the clean controlmulti_round: shared adoption, rounds 1 to 5seed_timing: holdout last vs firstmajority_pressure: one vs two seeded peersunanimity_break: unanimous wrong vs one dissentertrue_peer_control: wrong / correct peer adoptionorchestrator_failure: wrong peer poisons outputhierarchy_dominance: order-independent dominant agentscale_c: anchored vs generic conformcascade_C_flash: anchored vs genericpush_c: conform in any conditionbreak_it: contamination flag vs controlclean_a: flag ratecontamination_audit: flip rate when baseline wrong vs correct (Fisher)contamination_audit: options-only accuracy above chanceWith three adopting holdouts, every referee design flags exactly those three; the referee lane cannot discriminate between designs on this model and its precision/recall of 1.0 is a floor effect.
Reproducibility, from the committed script
experiments/medqa/cross_lineage_report.py --model nvidia/nemotron-3-super-120b-a12brecomputes both body-level claims from the committed summaries and caches:pvaluethe arm summaries report, BH at 0.05 within lineage: 43 contrasts per lineage, Gemini keeps 33, nemotron 18; the smallest dropped nemotron q is 0.071.Code
experiments/_lane.pyis the single dispatch for every text runner: key from the model id, backend, output cap, model-scoped output paths, andpaced_complete, which waits out a 429 and retries a dropped connection. The blind-metric lane now goes through it too, and its declared-only view uses the shared declaration detector (Gemini declared-only rows went from 0 to 98/100 on baseline as a result; no legacy field changed).is_localnever redirects a NIM house id to a local server. The vendor client runs withmax_retries=0and a settable timeout. Suite failing set is identical tomain.Not in this PR
Imaging arms (text-only model). SUPPORT2 and the MedMCQA cross-dataset arm (datasets not on the machine that ran this). The MIMIC-CXR text lane (its committed cohort is not reconstructible from the repository). Multi-model runners that pair two Gemini ids by design need a committee decision first.