MedMCQA standard-battery replication (tracking)

[duckyquang]

Tracking issue for the MedMCQA standard-battery replication. Each item mirrors an experiment from the finished MedQA/NIH template so results are apples-to-apples across datasets (see the workflow: replicate the standard battery on a new dataset).

Dataset adapter is already implemented (benchmaxxing/datasets/medmcqa.py, #112). These are replication, not new features: run the existing experiments on a MedMCQA manifest through the Gemini API, and write results under experiments/medmcqa/results/ in the same JSON/JSONL format as experiments/medqa/results/.

Foundational (run first)

• Solo shortcut susceptibility + noise floor (MedMCQA) #267 Solo shortcut susceptibility + noise floor (MedMCQA)
• Cascade baseline: bare and anchored seed (MedMCQA) #268 Cascade baseline: bare and anchored seed (MedMCQA)

Seed shape

• Plausible vs implausible planted distractor (MedMCQA) #269 Plausible vs implausible planted distractor (MedMCQA)
• Rationale validity: bare vs valid-but-wrong vs named-fallacy seed (MedMCQA) #270 Rationale validity: bare vs valid-but-wrong vs named-fallacy seed (MedMCQA)
• Seed confidence: hedged vs confident wrong peer (MedMCQA) #271 Seed confidence: hedged vs confident wrong peer (MedMCQA)
• Attributed-tier identity of the seed (MedMCQA) #272 Attributed-tier identity of the seed (MedMCQA)
• Cue dose-response: text magnitude sweep (MedMCQA) #273 Cue dose-response: text magnitude sweep (MedMCQA)
• Text cue types: primacy/position and negation/qualifier (MedMCQA) #274 Text cue types: primacy/position and negation/qualifier (MedMCQA)

Social pressure / committee structure

• Majority-pressure (Asch) variant (MedMCQA) #275 Majority-pressure (Asch) variant (MedMCQA)
• Committee-size sweep: dilution and proportional pressure (MedMCQA) #276 Committee-size sweep: dilution and proportional pressure (MedMCQA)
• Dissenter / unanimity break (MedMCQA) #277 Dissenter / unanimity break (MedMCQA)
• Deliberation framing crossed with the anchored seed (MedMCQA) #278 Deliberation framing crossed with the anchored seed (MedMCQA)
• Super-additivity 2x2: system flag AND anchored senior peer (MedMCQA) #279 Super-additivity 2x2: system flag AND anchored senior peer (MedMCQA)

Hierarchy / orchestrator

• Hierarchy dominance via order-permutation committees (MedMCQA) #280 Hierarchy dominance via order-permutation committees (MedMCQA)
• Wrong orchestrator vs wrong peer (MedMCQA) #281 Wrong orchestrator vs wrong peer (MedMCQA)
• Authority gradient ladder (MedMCQA) #282 Authority gradient ladder (MedMCQA)
• Leader-as-auditor remediation prompt (MedMCQA) #283 Leader-as-auditor remediation prompt (MedMCQA)

Controls & robustness

• True-peer negative control: plant the CORRECT answer (MedMCQA) #284 True-peer negative control: plant the CORRECT answer (MedMCQA)
• Test-awareness manipulation on the cascade (MedMCQA) #285 Test-awareness manipulation on the cascade (MedMCQA)
• Temperature sensitivity + run-to-run reliability (MedMCQA) #286 Temperature sensitivity + run-to-run reliability (MedMCQA)
• Onset + contagion/deference battery over transcripts (MedMCQA) #287 Onset + contagion/deference battery over transcripts (MedMCQA)

Break-it arms

• Break-it A: contaminated shared context / system pre-screen flag (MedMCQA) #288 Break-it A: contaminated shared context / system pre-screen flag (MedMCQA)
• Break-it C: case-anchored plausible seed vs generic (MedMCQA) #289 Break-it C: case-anchored plausible seed vs generic (MedMCQA)
• Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290 Break-it D: blind-metric decoy rewards the longest option (MedMCQA)

Related: #129 (MedQA vs MedMCQA cross-dataset consistency, @Agastya191) is adjacent - coordinate to avoid duplicate cascade runs.

Referee (was missing; the named contribution)

• Referee detection: naive gate vs targeted vs deployable (MedMCQA) #322 Referee detection: naive gate vs targeted vs deployable

[sebasmos]

Heads up: the solo-cue cell is already shown consistent with MedQA via #266. The full battery (#267-#290) hasn't started yet though, that's the sprint priority right now over more infra PRs, deadline's the 29th.

[sebasmos]

Verification ask before submission, not a re-run request. @Agastya191 and I have both recomputed your arms from the raw rows and found no arithmetic disagreement, so this is purely about whether anyone else can check them.

Every MedQA arm replays keylessly from a committed cache. No MedMCQA arm does: .gitignore:21 excludes *_cache.jsonl, so all 18 MedMCQA caches are absent while all 18 MedQA equivalents are tracked. Reproducing super_additivity's committed 0.70 / 0.7417 / 0.85 costs roughly 480 billed calls, where the identical MedQA command costs zero. MedMCQA is public, so unlike the MIMIC lanes there is no DUA reason for it.

Could you commit the caches with a ! exception in the gitignore? That single change makes the whole lane verifiable by anyone. Also worth adding a runner or a README with the exact commands, since experiments/medmcqa/ currently contains only results/ and no invocation is recorded for any of the 26 arms.

[duckyquang]

Status update on the battery:

• All 22 sub-arms are done and now closed (Plausible vs implausible planted distractor (MedMCQA) #269–Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290), each pointing at its results/RESULTS.md section. Majority-pressure (Asch) variant (MedMCQA) #275/Hierarchy dominance via order-permutation committees (MedMCQA) #280/Wrong orchestrator vs wrong peer (MedMCQA) #281/Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290 closed with their documented null/degenerate findings — those are design-inherent at temp 0, so re-running won't change them.
• The reproducibility gate you flagged (@sebasmos) is addressed in chore(#291): commit MedMCQA committee-arm caches + repro README #415: the 20 committee-arm *_cache.jsonl are now committed (they weren't before — call_cache.jsonl only covers the foundational arms), plus a experiments/medmcqa/README.md recording the stage command and per-arm invocations. Every committee arm now replays keylessly at zero API calls.

Leaving this tracker open until #415 merges; I'll tick the checklist and close it then.

[duckyquang]

The full battery is done and reproducible, so closing the tracker.

• All 24 sub-arms are closed: foundational + break-it (Solo shortcut susceptibility + noise floor (MedMCQA) #267/Cascade baseline: bare and anchored seed (MedMCQA) #268/Break-it A: contaminated shared context / system pre-screen flag (MedMCQA) #288/Break-it C: case-anchored plausible seed vs generic (MedMCQA) #289/Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290) via results: MedMCQA foundational arms — solo, cascade, break-it A + C #349/results: MedMCQA committee battery — seed shape, social pressure, hierarchy, controls, referee + corrected family #368; seed shape / social pressure / hierarchy / controls (Plausible vs implausible planted distractor (MedMCQA) #269-Onset + contagion/deference battery over transcripts (MedMCQA) #287) and referee (Referee detection: naive gate vs targeted vs deployable (MedMCQA) #322) via results: MedMCQA committee battery — seed shape, social pressure, hierarchy, controls, referee + corrected family #368.
• Written up in experiments/medmcqa/results/RESULTS.md, with Majority-pressure (Asch) variant (MedMCQA) #275/Hierarchy dominance via order-permutation committees (MedMCQA) #280/Wrong orchestrator vs wrong peer (MedMCQA) #281/Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290 documented as design-inherent nulls at temp 0.
• Committee-arm caches committed in chore(#291): commit MedMCQA committee-arm caches + repro README #415, so every arm replays keylessly at zero API calls, plus experiments/medmcqa/README.md records the stage + per-arm commands.