Text lineage: openai/gpt-oss-120b - #427
Draft
sebasmos wants to merge 20 commits into
Draft
Conversation
…rom #421) The --model flag, per-model key and backend dispatch, the shared experiments/_lane.py with all nineteen single-model text runners ported to it, and second-vendor pacing, taken from feat/text-lane-model-flag without that branch's result files. The Gemini blind-metric comparator is extended to n=100 so a second lineage runs the same 100 cases. This commit is the part of #421 a per-model results branch depends on; it carries no second-lineage results of its own.
…dpoint The dispatch sent every non-Gemini id to NVIDIA NIM, so an open-weights arm could only run through a vendor API and inherited a rate limit that does not apply to it. Setting BENCHMAXXING_LOCAL_BASE_URL now points the OpenAI-compatible backend at that server, drops the key lookup, since no local server checks one, and switches pacing off, since the interval exists only to respect a vendor request ceiling. Gemini and DeepSeek ids keep their vendor routing whatever the variable is set to, so it cannot redirect a committed comparator arm to a different model behind the same id. The blind-metric lane carries its own copy of the key and backend dispatch, so it reads the same variable on the same terms. Prompts, parsers and the reasoning cap are untouched, so prompts stay byte comparable across lineages. (cherry picked from commit f151941)
openai/gpt-oss-120b on the same manifest cases as the Gemini and nemotron arms: the four MedQA runners at n=120 and blind-metric at n=100 plus the matched 40-case cohort. Served on four H100 cards with vLLM, so the arm costs no vendor calls and the whole run is one pass with no rate limiting. The model returns the bare option letter in content and keeps its reasoning out of it, so nothing truncated is ever scored. Caches are committed, force added past the ignore rule as the Gemini cache is, so every number replays with no key and no server. Four allowlist entries cover the two blind-metric files, one definitional and one empirical per file, each stating what was read. (cherry picked from commit 423ac8c)
live_peer_organic.py passed (model, prompt) to the shared cache, whose signature is complete(prompt, model=None), so the question text went out as the model id and every call failed with 404; and its --model never reached the committee, whose holdout was bound to the Gemini constant. The holdout is now the requested model and the two flash peers read their answers from the committed cache, so a new holdout faces exactly the board the Gemini holdout saw. temperature_sensitivity.py called the shared cache with a temperature and sample index it does not take, raising TypeError on the first call. It gets a draw-aware cache on the shared dispatch, keyed as the committed Gemini sweep was written, so that arm still replays with no calls. Both default-model arms replay row-identical from their committed caches. (cherry picked from commit e46c361)
The runner imported the Gemini-only cache from referee_threshold.py, so the floor could not be measured on a second model. It now carries a draw-aware cache on the shared text-lane dispatch with the same key, so the committed Gemini arm replays with no calls and its summary stays byte identical, and any model the text lane can address runs through it. (cherry picked from commit 3c3fae6)
…nd the live-peer board on gpt-oss-120b All fourteen single-model MedQA runners plus the deliberation channel arm, the referee self-inconsistency floor and the live-peer organic board, on the same manifest cases as the Gemini and nemotron arms, 8,092 cached calls in all. The channel runner gains a gpt-oss branch because the model has no thinking switch: enable_thinking is silently ignored and reasoning_effort only budgets the hidden channel, so the rows record content length to make that visible. Twenty-two allowlist entries cover the constant, duplicate and rounded columns, each stating what was read. (cherry picked from commit d076c52)
One branch, one new model against the committed gemini-2.5-flash-lite baseline on the same 120 MedQA cases. Eighteen MedQA arms, blind metric at n=40 and n=100, and the referee self-inconsistency floor. The branch carries the shared model dispatch it needs to run at all, because main has no experiments/_lane.py yet: key and backend resolution, the locally served OpenAI-compatible path, output caps, pacing and transient recovery, plus the runner ports that go through it. No other lineage's result files are here. Once that dispatch lands on main this branch reduces to its own arms.
…d dispatch break_it, clean_a, push_c, scale_c, hierarchy_dominance, hierarchy_temp, majority_pressure, orchestrator_failure, seed_timing, true_peer_control, unanimity_break and reproduce each kept a private cache that constructed GeminiBackend directly, so none could run for a second model. Each keeps its own cache class and key format, and the committed Gemini arms replay unchanged: the shared dispatch returns the identical GeminiBackend for a Gemini id, and with the default model the runner still calls its own key lookup and writes to its committed paths. When --model names another model, _lane.rebind_models puts it in every Gemini seat the module names (HOLDOUT, MODELS, TIERS, MEMBERS, COMMITTEE), so a committee runner compares that model's committee against Gemini's rather than mixing lineages, and a list of tiers collapses to one entry per distinct model so no runner calls the same model twice for one seat. The key comes from the shared lookup and the paths are model-scoped. 28 tests.
…and MIMIC-CXR text runners a --model flag Seventeen more runners constructed GeminiBackend directly inside a private cache, so none could run for a second model: the four referee runners, cascade/multi_round, contamination_audit, cascade_C_flash, the five SUPPORT2 runners through their shared _common cache, and the five MIMIC-CXR text runners that the earlier port left behind. Same terms as the MedQA port: each keeps its own cache class and key format so the committed Gemini arm replays unchanged (checked against main for every runner whose cache covers the manifest), every Gemini seat including a judge is rebound to the requested model, and output and cache paths are model-scoped. A --cache default that named the committed file now defaults to the scoped file, so a second model can never append to a committed Gemini cache. 52 tests.
The hard-case runners read a per-record solo file that reproduce.py computed but never wrote; the committed Gemini copy came from an earlier tool. Written only for a non-default model so a replay of the default arm cannot rewrite the committed file.
… port for every Gemini-only runner feat/gpt-oss-120b-lineage and this branch carried the same 62 gpt-oss-120b result files, byte-identical, with the machinery split between them. This branch had the newer transport (paced_complete, transient recovery, the parameterised client timeout, declared_only_summary); the other had the --model port for the thirty runners that hardcoded a Gemini id (referee, cascade, contamination, model-dependence, SUPPORT2, MIMIC-CXR text and the twelve Gemini-only MedQA runners) with rebind_models and the two tests that pin every one of them to the shared dispatch. Taken here: this branch's version of every shared file, plus rebind_models and the thirty ported runners and their tests from the other branch. The Gemini n=100 blind-metric comparator comes in at the regenerated version that carries named_rubric_when_declared_drifted, same rows, no pre-existing value changed. One exemption entry added for that file's constant column. Suite: failing set identical to main (the two platform-scoped golden checksums), 1345 passed.
…eplays on this branch The description cites Gemini's none/hidden/open adoption next to gpt-oss's, but the Gemini result, cache and summary for that arm were not on this branch. Same files as the nemotron branch, byte-identical, with their nine guard entries.
break_it, clean_a, push_c, scale_c, reproduce and the MIMIC-CXR text runners with a fixed cache name kept their Gemini cache at out / name, so the MedMCQA battery (the MedQA runners with --out experiments/medmcqa/results) found it beside the results. The port had derived the model-scoped path from a hard-coded MedQA directory instead, which would have put a second model's MedMCQA cache in the MedQA lane. The scoped cache now follows --out exactly as the Gemini one does; runners with a --cache flag are unchanged.
break_it, clean_a, push_c, scale_c, hierarchy_dominance, hierarchy_temp, majority_pressure, orchestrator_failure, seed_timing, true_peer_control, unanimity_break and reproduce, on the same manifest and at the n each committed Gemini summary reports, with openai/gpt-oss-120b in every seat. Fixed-cohort arms carry the same case ids as the Gemini rows; the arms that filter cases by the model's own answers (clean-correct, solo-wrong, probed hard cases) differ in n by construction. 3585 cached calls in 74 files, every arm replays with no key and no server. The hard-case runners read the solo records reproduce.py now writes for this model.
… gpt-oss-120b referee_deployable, referee_judge, referee_threshold and referee_requery_design at n=40, cascade/multi_round at n=40, contamination_audit on this model's own 100-case solo set, and cascade_C_flash on its probed hard cases; the judge and orchestrator seats are openai/gpt-oss-120b as well. 1947 cached calls in 19 files, every arm replays with no key and no server.
solo, cascade, cascade_strength, referee and referee_judge at n=120 on the committed SUPPORT2 manifest, with openai/gpt-oss-120b in every seat including the judge. The committed Gemini arms replay at zero calls with identical rows first; the gpt-oss case ids match Gemini's case for case, and the referee clean-control block is 120 rows against Gemini's 115 because this model never abstains. 2760 cached calls in 12 files.
The 500-row MedMCQA manifest rebuilt from the public validation split as the lane README prescribes; the committed Gemini caches replay at zero calls with identical case ids on 20 of 23 arms, and the three that miss (orchestrator_failure, clean_a, scale_c) miss identically on main. All 23 API-calling arms run with openai/gpt-oss-120b in every seat at the committed n, into experiments/medmcqa/results/openai_gpt-oss-120b/ with the caches beside them. 9880 cached calls in 68 files.
…de and SUPPORT2 files Fifty-three entries, each stating what was read: the definitional columns cite the same code the Gemini and MedMCQA entries for those columns cite; the empirical zeros (no abstention on any SUPPORT2 board, no adoption with a correct dissenter) name the comparator rows that show the column is live; the duplicate flags in the referee family carry the open-defect wording of #374 and the forced-direction tests on SUPPORT2 that of #391; every rounded p is recomputed exactly from its discordant pair.
…eed_confidence at n=100 reproduce.py --stage all on the MedMCQA manifest with openai/gpt-oss-120b in every seat: the 100-case seeded solo set is the same as the committed Gemini one (seed 0 over the same 500 rows), flip rate 0.12 with a 0.067 uncached noise floor, and the 20-case cascade. Those two stages have no summary file, which is how the battery missed them. seed_confidence on MedQA is re-emitted at the n the committed Gemini arm reports (100, a prefix of the 120 already cached) with no new calls.
sebasmos
force-pushed
the
feat/gpt-oss-120b-text
branch
from
September 9, 2026 00:19
8056b38 to
6f995c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Text lineage:
openai/gpt-oss-120bLocally served, same MedQA manifest and same 120 cases as the committed
gemini-2.5-flash-litearms, temperature 0. 29 MedQA arms, blind metric at n=40 and n=100, the referee floor, and every other text lane the repo can run (MedMCQA, SUPPORT2, referee, cascade, contamination, model dependence; table at the end). Every number replays from a fresh clone with no API calls. Draft.Unseeded accuracy on the shared cases: Gemini 101/120 in every arm; gpt-oss between 110/120 and 115/120 depending on the arm (111/120 on the cue-types arm). The second figure varies because a temperature-0 answer from this model is not reproducible across sessions, so every contrast below is within-arm.
blind_metricblind_metrictext_cue_typestext_cue_typesdose_responsedose_responseauthority_ladderauthority_laddertest_awarenesscommittee_size_sweepplausible_distractorlive_peer_organicsuper_additivitytemperature_sensitivitydeliberation_channelreferee_self_inconsistencyAdoption is roughly an order of magnitude below Gemini on almost every arm, and it is not a capability floor: this model answers more of the unseeded questions correctly. The ordering within arms is preserved, so the cues rank the same way at very different amplitude.
The one place it does not stay low is authority provenance. An unnamed clinical guideline takes adoption from 0.033 under an automated-system attribution to 0.800, a twenty-four-fold jump, while a named senior attending only reaches 0.100. Deference here is to the kind of source, not to seniority.
Two arms show the same value in both lineages: following a wrong organic peer consensus is 7/8 in each, and both are sub-additive. Note that the set of peer answers is identical across lineages by construction, since the peers answer before the holdout and never see it.
Code
Main has no
experiments/_lane.py, so this branch carries the shared model dispatch it needs to run at all: key and backend resolution, the locally served OpenAI-compatible path, output caps, call pacing and transient recovery, plus the runner ports that go through it: the eighteen ablation arms and the referee floor, and the thirty runners that hardcoded a Gemini id (referee, cascade, contamination, model-dependence, SUPPORT2, MIMIC-CXR text and the twelve Gemini-only MedQA runners), whichrebind_modelspoints at the requested model. Two tests pin every one of them to the shared dispatch. Once that dispatch is on main this branch reduces to its own arms.Scope
One branch, one new model against the Gemini baseline. No other lineage's results are here. Imaging is not covered: this model is text-only. The multi-model runners seat this model in every Gemini seat (
rebind_models), so their rows compare an all-gpt-oss committee with the all-Gemini one; if a mixed committee is wanted instead, that is a one-line change in the runner and a re-run.Other lanes
Every Gemini text experiment the repo can still run now has a gpt-oss arm: 29 MedQA summaries, 23 on the MedMCQA cohort (500-row manifest rebuilt as the lane README prescribes; the committed Gemini caches replay at zero calls on 20 of 23 arms and the three misses are identical on main), 5 referee, 5 SUPPORT2, plus cascade, contamination and model-dependence. Committee runners seat this model in every seat, judge and orchestrator included, so each row pairs an all-gpt-oss committee against Gemini's all-Gemini one.
text_cue_typesdose_responseauthority_laddertest_awarenesscommittee_size_sweepplausible_distractortemperature_sensitivityreproduce (solo)majority_pressureunanimity_breaktrue_peer_controlorchestrator_failurehierarchy_dominanceseed_timingpush_cmulti_roundrun_cross_datasetcontamination_auditreferee_deployablereferee_judgesupport2_solosupport2_cascadesupport2_refereesupport2_referee_judgeThe low amplitude is MedQA-specific. On SUPPORT2 the shared board carries the wrong prognosis on 56% of cases (Gemini 100%), and on MedMCQA the baseline cue reaches 0.317 with the guideline rung at 0.983, above Gemini's 0.917. This model never abstains on the SUPPORT2 board, so its clean-control block is 120 rows to Gemini's 115.
The cue pilot needed a port before it could run at all: it resolved a Gemini key and constructed a Gemini backend directly, so no second model could reach it. On the default model the path is unchanged, and its pre-existing cache miss reproduces on
main. Its cue rates are dataset-dependent on this model (mean spread 0.147 against 0.047), unlike the Gemini pilot where no cue differed across the two cohorts.Not covered, and why: the imaging lanes (text-only model); the MIMIC-CXR text lane, whose 7 runners need the 633 committed case ids that the repo's history rewrite removed and fail on main with
CaseIndexErrorfor any model; and the summaries that make no model calls (stats_reconciliation,cross_lane_reconciliation, the derived cascade summaries).