Third lineage for the cross-lineage comparison: gpt-oss-120b (text) and Qwen2.5-VL-72B (imaging) - #423
Merged
Conversation
…dpoint The dispatch sent every non-Gemini id to NVIDIA NIM, so an open-weights arm could only run through a vendor API and inherited a rate limit that does not apply to it. Setting BENCHMAXXING_LOCAL_BASE_URL now points the OpenAI-compatible backend at that server, drops the key lookup, since no local server checks one, and switches pacing off, since the interval exists only to respect a vendor request ceiling. Gemini and DeepSeek ids keep their vendor routing whatever the variable is set to, so it cannot redirect a committed comparator arm to a different model behind the same id. The blind-metric lane carries its own copy of the key and backend dispatch, so it reads the same variable on the same terms. Prompts, parsers and the reasoning cap are untouched, so prompts stay byte comparable across lineages.
openai/gpt-oss-120b on the same manifest cases as the Gemini and nemotron arms: the four MedQA runners at n=120 and blind-metric at n=100 plus the paper-matched 40. Served on four H100 cards with vLLM, so the arm costs no vendor calls and the whole run is one pass with no rate limiting. The model returns the bare option letter in content and keeps its reasoning out of it, so nothing truncated is ever scored. Caches are committed, force added past the ignore rule as the Gemini cache is, so every number replays with no key and no server. Four allowlist entries cover the two blind-metric files, one definitional and one empirical per file, each stating what was read.
live_peer_organic.py passed (model, prompt) to the shared cache, whose signature is complete(prompt, model=None), so the question text went out as the model id and every call failed with 404; and its --model never reached the committee, whose holdout was bound to the Gemini constant. The holdout is now the requested model and the two flash peers read their answers from the committed cache, so a new holdout faces exactly the board the paper holdout saw. temperature_sensitivity.py called the shared cache with a temperature and sample index it does not take, raising TypeError on the first call. It gets a draw-aware cache on the shared dispatch, keyed as the committed Gemini sweep was written, so that arm still replays with no calls. Both default-model arms replay row-identical from their committed caches.
The runner imported the Gemini-only cache from referee_threshold.py, so the floor could not be measured on a second model. It now carries a draw-aware cache on the shared text-lane dispatch with the same key, so the committed Gemini arm replays with no calls and its summary stays byte identical, and any model the text lane can address runs through it.
Qwen/Qwen2.5-VL-72B-Instruct on the 35 CheXpert cases of nih_match_35.csv, served with vLLM on four cards, through the --model version of the runner from #416 plus the same local-server routing the text lane uses. Blind decoy uptake is 14 of 35 off a zero baseline and the test-aware prime suppresses it on every case. The images come from a public mirror by exact CheXpert-v1.0-small path. Their bytes do not reproduce the committed Gemini cache keys, so the pixels are re-encoded relative to the originals and could not be verified byte identical; image_provenance.json records the sha256, size and dimensions of each file used.
…nd the live-peer board on gpt-oss All fourteen single-model MedQA runners plus the deliberation channel arm, the referee self-inconsistency floor and the live-peer organic board, on the same manifest cases as the Gemini and nemotron arms, 8,092 cached calls in all. The channel runner gains a gpt-oss branch because the model has no thinking switch: enable_thinking is silently ignored and reasoning_effort only budgets the hidden channel, so the rows record content length to make that visible. Twenty-two allowlist entries cover the constant, duplicate and rounded columns, each stating what was read; the imaging entries for the previous commit are included here.
sebasmos
marked this pull request as ready for review
September 7, 2026 14:41
sebasmos
requested review from
Agastya191,
MohShahin,
PeterLi-jpg,
Yehudha-kennedy,
aadityapanchal7,
amarzullo24,
armaanvgrewal,
arshnoor08,
azzy-chemE,
duckyquang,
felipeocampoos,
maximinl and
renato-umeton
September 7, 2026 14:41
sebasmos
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third text lineage for the cross-lineage comparison in #421:
openai/gpt-oss-120b, served locally with vLLM, on the samemanifest_test.csvcases as the Gemini and nemotron arms. Plus Armaan's referee floor (#418) and Kennedy's imaging blind metric (#416) on a third vision lineage,Qwen/Qwen2.5-VL-72B-Instruct. All caches committed; every number replays keyless.Headline arms (n=120; blind metric n=100)
Five of ten paired tests reproduce in all three lineages: negation cue, dose monotonicity, the inverted U at emphatic, blind-metric drift off a zero baseline, and test-aware suppression of it. Amplitude on gpt-oss is about a tenth of Gemini, on the highest unseeded accuracy of the three.
Ablations (n=120)
The clinical-guideline rung is the one cell where all three lineages are within a factor of two: a model near-immune to planted human authority adopts a cited guideline on 96/120. gpt-oss follows real peers at Gemini's rate. gpt-oss has no thinking switch and answers in one character on every row, so the channel arm is flat on it; nemotron's crossover stays nemotron's.
Referee floor (#418): 1/40 unstable at temperature 0 on gpt-oss (Gemini 0/40).
Imaging blind metric (#416, n=35)
Direction replicates, suppression is complete. Images are the manifest's CheXpert-v1.0-small studies by exact path;
image_provenance.jsonrecords each file.Changes
BENCHMAXXING_LOCAL_BASE_URLroutes the text lane to a local OpenAI-compatible server; Gemini and DeepSeek ids keep vendor routing. 11 tests.live_peer_organic.py(model and prompt arguments swapped) andtemperature_sensitivity.py(draw-aware cache lost in the port).referee_self_inconsistency.pygiven--model. Committed Gemini arms replay row-identical. 4 tests.