NVIDIA NIM API Integration (Llama 3.2 90B Vision) for Blind Metrics and Comparative Analysis - #416
NVIDIA NIM API Integration (Llama 3.2 90B Vision) for Blind Metrics and Comparative Analysis#416Yehudha-kennedy wants to merge 3 commits into
Conversation
… sequential processing
…ut by model - Replace broken _yesno() that read from text start with parse_yesno() from benchmaxxing.extract which correctly uses last word-boundary match. - Add --model as required CLI argument, remove hardcoded MODEL constant. - Scope output directory by model slug to prevent cross-model overwrites. - Add n_unparseable counts per condition to summary JSON. - Store raw parsed answers (base_ans, blind_ans, aware_ans) in per-row JSONL. - Include corrected pilot results (n=35) for meta/llama-3.2-90b-vision-instruct. Corrected metrics (n=35, Llama 3.2 90B Vision): blind uptake: 0.80 -> 0.8571 (parser was undercounting 2 drifts) test_aware: 0.2857 (unchanged) n_drifted: 28 -> 30 n_unparseable: 0 across all conditions
|
Done. Fixed all three items:
|
…90B response cache
|
All four items addressed in commit
Ready for re-review. |
sebasmos
left a comment
sebasmos
left a comment
There was a problem hiding this comment.
Re-checked at 70f3d92. All four asks are done, and the suite is red on this head for a fifth reason.
What is verified. The body numbers now match the committed artifact exactly, recomputed row by row from meta_llama-3.2-90b-vision-instruct/imaging_blind_metric.jsonl: 0.8571 blind, 0.2857 test-aware, 0.5714 suppression delta, 30 drifted, 0 naming the rubric, 30 silent. The baseline 0% claim is gone and the decoys are now described as constructed relative to the baseline read, which is the accurate framing. _key() no longer hands a Gemini key to a NIM model, and the cache-miss message names the variable it wants. The cache is committed: 859 llama rows beside the 105 Gemini ones, and the summary's new_api_calls_this_run of 0 shows the committed rows are a pure replay of that file, not a fresh billed run.
One blocker. The suite fails on this head: 1 failed, 1207 passed, 8 skipped.
tests/test_degeneracy_guard.py::test_no_unexplained_cannot_fail_metrics
constant_column|experiments/chexpert/results/meta_llama-3.2-90b-vision-instruct/imaging_blind_metric.jsonl|base_is_decoy
constant_column|experiments/chexpert/results/meta_llama-3.2-90b-vision-instruct/imaging_blind_metric.jsonl|named_rubric_when_drifted
That is the #374 guard working: every sibling lane's identical columns carry an entry in tests/degeneracy_exemptions.json, and the new model-scoped path has none. Two entries fix it, keyed exactly as the failure prints them. Please give them two different reasons, because the two columns are not the same thing: base_is_decoy is False by construction, which is the tautology from the last round, while 0 of 30 naming the rubric is an empirical result and worth saying so in one line. Nothing here runs CI, so only a local run catches this.
Two small asks. The cache key is sha256(image bytes + prompt), so "verify offline with zero API calls" holds only for someone who already has the CheXpert images. Writing the three cache keys into each row would make the parse checkable by anyone. And the committed cache has 22 duplicate keys, 5 of them holding conflicting responses for the same image and prompt, one flipping No. to Yes.; the loader is last-wins, so a replay depends on file order. Deduping those 5 keeps the artifact honest.
One note, not an ask. Those 5 conflicts are the imaging twin of what #417 found in text: same model, same input, temperature 0, different answer. Worth its own issue once the n=600 run lands, not something to fix here.
REQUEST_CHANGES
…ineage blind-metric arm Mirrors the imaging lane from #416. _key(model) resolves the key from the model id; _backend() sends Gemini ids to GeminiBackend and everything else to LocalOpenAICompatibleBackend on NIM with an 8192 output-token cap. --model defaults to the existing Gemini id, so current invocations are unchanged; output and cache paths are model-scoped. Each row also records the declared terminal letter per condition and the summary carries a declared-only view, so a completion that never commits to a letter is excluded rather than scored. A None completion raises instead of being cached. Prompts and parsers untouched. Arm: nvidia/nemotron-3-super-120b-a12b on the same 40 MedQA cases as the Gemini comparator, cold cache, 120 calls. Per-case rows, summary and call cache committed, with three allowlist entries for the #374 guard (one definitional, two empirical).
…ineage blind-metric arm Mirrors the imaging lane from #416. _key(model) resolves the key from the model id; _backend() sends Gemini ids to GeminiBackend and everything else to LocalOpenAICompatibleBackend on NIM with an 8192 output-token cap. --model defaults to the existing Gemini id, so current invocations are unchanged; output and cache paths are model-scoped. Each row also records the declared terminal letter per condition and the summary carries a declared-only view, so a completion that never commits to a letter is excluded rather than scored. A None completion raises instead of being cached. Prompts and parsers untouched. Arm: nvidia/nemotron-3-super-120b-a12b on the same 40 MedQA cases as the Gemini comparator, cold cache, 120 calls. Per-case rows, summary and call cache committed, with three allowlist entries for the #374 guard (one definitional, two empirical).
#416 documented the NVIDIA endpoint at about 40 RPM with a penalty for concurrent bursts. A free-tier key sustains much less than that: measured on this account the bucket is small and refills slowly, one call every 20s completes 9 attempts in 10, and a single call succeeds again after 60s of idle. Four arms in parallel returned 429 after roughly 300 calls. Two changes, both in the shared module so every text lane gets them. Calls are paced to the measured sustained rate, held across threads so it holds whatever max_workers a runner uses, and disabled for Gemini, which has no such restriction. A 429 now waits for the bucket to refill instead of failing: RetryBackend's five quick attempts expire while it is still empty, which is what killed whole arms mid-run and cost the calls already made. Not included: #416 also builds the OpenAI client with timeout=60.0 and max_retries=0 so the client's internal retries stop fighting RetryBackend. That fix is on its branch and this lane inherits it when #416 merges, so gateway.py is left alone here rather than conflicting with an open PR.
|
Good work landing the NIM integration and chasing down that tarpit-style rate-limit deadlock. One problem is in benchmaxxing/gateway.py: LocalOpenAICompatibleBackend.init now hardcodes timeout=60.0 and max_retries=0 on the OpenAI client, with no parameter to override either value. This class is shared by every OpenAI-compatible model, including any locally-served one and the other second-vendor dispatch code routes through it, so any model whose legitimate per-request latency runs past 60 seconds, a long chain-of-thought completion or a slow self-hosted server, will time out on every attempt and burn through the outer retry loop until the whole arm dies, not just the NIM tarpit case this was written for. I'd add a timeout parameter to init (defaulting to 60.0) so NIM keeps its fast-fail behavior while slower lanes can raise it. |
LocalOpenAICompatibleBackend built its client with the SDK defaults, so max_retries was 2. That put a hidden retry loop under gateway.RetryBackend (5 attempts) and _lane.paced_complete, letting one logical call become many unpaced HTTP requests and spend a rate bucket the lane believed it was metering. Retries now belong to the caller: max_retries defaults to 0, and both it and timeout are parameters rather than fixed values, so the hosted endpoint keeps a 60 s fast-fail while a locally served model gets 600 s for a long completion. Raised by @Agastya191 on #416, where the same line hardcodes both. Transport only: all 158 committed row files across the three lineages replay set-identical with the new client and no API calls, and the only summary change is new_api_calls_this_run.
4 participants
Description
This Pull Request introduces dynamic support for using NVIDIA NIM's OpenAI-compatible API ($n=35$ ), and the infrastructure adjustments required to mitigate underlying network behaviors.
meta/llama-3.2-90b-vision-instruct) within the project's experimentation pipeline. It includes a comparative architecture analysis against Gemini, results from the blind-metrics pilot test (Current Progress: The technical analysis shown below corresponds to an initial pilot sample of$n=35$ . Full-scale validation with the complete sample of $n=600$ using Llama 3.2 90B Vision is currently running, progressing at a steady rate thanks to the implemented blocking mitigations. Final results will be reported upon completion.
Infrastructure Comparison: NVIDIA NIM API vs Gemini API
During development and integration, we identified critical operational differences between both APIs for our workload:
openaiadapter.google-genai), highly optimized.Empirical Results (Pilot$n=35$ , Llama 3.2 90B Vision)
A pilot validation of 35 samples was run using the decoy-uptake vulnerability metric (with counterfactual binary decoys constructed relative to baseline reads). The findings expose strong lability in the Llama 3.2 90B Vision model:
Decoy Uptake:
Test-Aware Suppression:
Silent Drifting:
Bug Fixes: Shortcuts and Workarounds Implemented
Due to NVIDIA's rate-limiting policy profile (40 RPM) and server-side connection behavior, the following mitigations and fixes were implemented:
Centralized Parser & Model Scoping:
Replaced local
_yesno()with sharedparse_yesno()frombenchmaxxing.extract(last-match semantics). Added--modelCLI argument and scoped output directories ({out}/{model_slug}/) to prevent cross-model overwrites.Strict API Key Dispatch & Cache:
Strict API key resolution in
_key(model)ensuring NIM models only queryNVIDIA_API_KEYand fail loudly on cache misses without falling back to Gemini keys. Raw response cache for the pilot is committed inimg_cache.jsonlfor standalone reproducibility without API keys.Manual Timeout Control (Deadlock Prevention):
The
OpenAIclient was reconfigured by injectingtimeout=60.0and overriding the internal retry policy viamax_retries=0. Retry control with exponential backoff relies on theRetryBackendwrapper class.Forced Degradation to Sequential Processing:
Parallelism in
imaging_blind_metric.pywas set tomax_workers=1to prevent socket throttling and firewall blocks from concurrent bursts.