gpt-oss-120b as a second lineage on the text lane, served locally - #425
Closed
sebasmos wants to merge 9 commits into
Closed
gpt-oss-120b as a second lineage on the text lane, served locally#425sebasmos wants to merge 9 commits into
sebasmos wants to merge 9 commits into
Conversation
…rom #421) The --model flag, per-model key and backend dispatch, the shared experiments/_lane.py with all nineteen single-model text runners ported to it, and second-vendor pacing, taken from feat/text-lane-model-flag without that branch's result files. The Gemini blind-metric comparator is extended to n=100 so a second lineage runs the same 100 cases. This commit is the part of #421 a per-model results branch depends on; it carries no second-lineage results of its own.
…dpoint The dispatch sent every non-Gemini id to NVIDIA NIM, so an open-weights arm could only run through a vendor API and inherited a rate limit that does not apply to it. Setting BENCHMAXXING_LOCAL_BASE_URL now points the OpenAI-compatible backend at that server, drops the key lookup, since no local server checks one, and switches pacing off, since the interval exists only to respect a vendor request ceiling. Gemini and DeepSeek ids keep their vendor routing whatever the variable is set to, so it cannot redirect a committed comparator arm to a different model behind the same id. The blind-metric lane carries its own copy of the key and backend dispatch, so it reads the same variable on the same terms. Prompts, parsers and the reasoning cap are untouched, so prompts stay byte comparable across lineages. (cherry picked from commit f151941)
openai/gpt-oss-120b on the same manifest cases as the Gemini and nemotron arms: the four MedQA runners at n=120 and blind-metric at n=100 plus the matched 40-case cohort. Served on four H100 cards with vLLM, so the arm costs no vendor calls and the whole run is one pass with no rate limiting. The model returns the bare option letter in content and keeps its reasoning out of it, so nothing truncated is ever scored. Caches are committed, force added past the ignore rule as the Gemini cache is, so every number replays with no key and no server. Four allowlist entries cover the two blind-metric files, one definitional and one empirical per file, each stating what was read. (cherry picked from commit 423ac8c)
live_peer_organic.py passed (model, prompt) to the shared cache, whose signature is complete(prompt, model=None), so the question text went out as the model id and every call failed with 404; and its --model never reached the committee, whose holdout was bound to the Gemini constant. The holdout is now the requested model and the two flash peers read their answers from the committed cache, so a new holdout faces exactly the board the Gemini holdout saw. temperature_sensitivity.py called the shared cache with a temperature and sample index it does not take, raising TypeError on the first call. It gets a draw-aware cache on the shared dispatch, keyed as the committed Gemini sweep was written, so that arm still replays with no calls. Both default-model arms replay row-identical from their committed caches. (cherry picked from commit e46c361)
The runner imported the Gemini-only cache from referee_threshold.py, so the floor could not be measured on a second model. It now carries a draw-aware cache on the shared text-lane dispatch with the same key, so the committed Gemini arm replays with no calls and its summary stays byte identical, and any model the text lane can address runs through it. (cherry picked from commit 3c3fae6)
…nd the live-peer board on gpt-oss-120b All fourteen single-model MedQA runners plus the deliberation channel arm, the referee self-inconsistency floor and the live-peer organic board, on the same manifest cases as the Gemini and nemotron arms, 8,092 cached calls in all. The channel runner gains a gpt-oss branch because the model has no thinking switch: enable_thinking is silently ignored and reasoning_effort only budgets the hidden channel, so the rows record content length to make that visible. Twenty-two allowlist entries cover the constant, duplicate and rounded columns, each stating what was read. (cherry picked from commit d076c52)
…d dispatch break_it, clean_a, push_c, scale_c, hierarchy_dominance, hierarchy_temp, majority_pressure, orchestrator_failure, seed_timing, true_peer_control, unanimity_break and reproduce each kept a private cache that constructed GeminiBackend directly, so none could run for a second model. Each keeps its own cache class and key format, and the committed Gemini arms replay unchanged: the shared dispatch returns the identical GeminiBackend for a Gemini id, and with the default model the runner still calls its own key lookup and writes to its committed paths. When --model names another model, _lane.rebind_models puts it in every Gemini seat the module names (HOLDOUT, MODELS, TIERS, MEMBERS, COMMITTEE), so a committee runner compares that model's committee against Gemini's rather than mixing lineages, and a list of tiers collapses to one entry per distinct model so no runner calls the same model twice for one seat. The key comes from the shared lookup and the paths are model-scoped. 28 tests.
sebasmos
force-pushed
the
feat/gpt-oss-120b-lineage
branch
from
September 8, 2026 08:31
db7869b to
2df3d32
Compare
…and MIMIC-CXR text runners a --model flag Seventeen more runners constructed GeminiBackend directly inside a private cache, so none could run for a second model: the four referee runners, cascade/multi_round, contamination_audit, cascade_C_flash, the five SUPPORT2 runners through their shared _common cache, and the five MIMIC-CXR text runners that the earlier port left behind. Same terms as the MedQA port: each keeps its own cache class and key format so the committed Gemini arm replays unchanged (checked against main for every runner whose cache covers the manifest), every Gemini seat including a judge is rebound to the requested model, and output and cache paths are model-scoped. A --cache default that named the committed file now defaults to the scoped file, so a second model can never append to a committed Gemini cache. 52 tests.
sebasmos
force-pushed
the
feat/gpt-oss-120b-lineage
branch
from
September 8, 2026 08:37
888adb8 to
87015df
Compare
The hard-case runners read a per-record solo file that reproduce.py computed but never wrote; the committed Gemini copy came from an earlier tool. Written only for a non-default model so a replay of the default arm cannot rewrite the committed file.
Member
Author
|
Closed as duplicate of #427, which now carries everything that was on this branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
openai/gpt-oss-120bagainst the committed Gemini 2.5 Flash-Lite arms: 18 MedQA arms at n=120, blind metric at n=100, referee floor at n=40, same cases, served locally with vLLM. 10,432 calls, all cached, replays keyless.Direction holds, amplitude is about a tenth of Gemini. Matches Gemini on organic peer consensus and the clinical-guideline cue. Reasoning-channel arm not testable on this model (no thinking switch).
First commit is the shared
--modeldispatch from #421 without its result files; the rest isBENCHMAXXING_LOCAL_BASE_URLfor a local server, fixes to three runners that could not run for a second model, results, caches and 22 allowlist entries. Suite failing set identical tomain.