Imaging lineage: Qwen2.5-VL-72B-Instruct blind metric - #426
Draft
sebasmos wants to merge 1 commit into
Draft
Conversation
One branch, one new model against the committed Gemini baseline, which for this arm is the gemini-2.5-flash run already on main over the same 35 studies. Blind-rubric drift 0.40 against Gemini's 0.86, both off a 0.00 baseline, and complete suppression under the test-aware prompt (0.00) where Gemini retains 0.17. The branch carries the shared model dispatch it needs to run at all, because main has no experiments/_lane.py yet: key and backend resolution, the locally served OpenAI-compatible path, output caps, pacing and transient recovery, plus the runner ports that go through it. None of the other lineages' result files are here. Once the text-lane dispatch lands on main this branch reduces to the imaging arm alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Imaging lineage:
Qwen/Qwen2.5-VL-72B-InstructLocally served, 35 CheXpert studies, the same set as the committed
gemini-2.5-flashimaging arm on main. Draft.Drift is under half Gemini's (14/35 studies against 30/35), and the test-aware prompt removes what remains entirely, taking uptake to 0.000 where Gemini retains 0.171. Absolute suppression is therefore smaller than Gemini's (0.400 against 0.686) because there was less to suppress; as a fraction of its own drift it is complete. Neither model named the rubric on any drifted study.
Reading the direction matters here as much as the rate: the decoy is the opposite of each model's own baseline read, so drift is only deference if the baseline was right. Any comparison of two models' uptake numbers should be read next to their baseline accuracy on the same studies.
Code
Main has no
experiments/_lane.py, so this branch carries the shared model dispatch it needs to run at all: key and backend resolution, the locally served OpenAI-compatible path, output caps, call pacing and transient recovery, plus the runner ports that go through it. Once that dispatch is on main this branch reduces to the imaging arm alone.Scope
One branch, one new model against the Gemini baseline. No other lineage's results are here. The MedQA text battery on this model is in progress and will be added to this branch, so the same arms exist for it as for the text lineages.