[Quality] GPQA-Diamond smoke run (10 samples) - #36
Conversation
Add GPQA-Diamond, MMLU-Pro, HLE, LiveCodeBench, BFCL v4, SciCode, SWE-bench Pro, and DeepSWE as new quality-eval scenario types. - 8 adapted benchmark scripts under benchmarks/single_node/quality/ - launch_quality-eval.sh: auto-setup (uv, venvs, repo clones), persistent cache at ~/.quality-eval-cache, result collection - launch_bench-client.sh: route SCENARIO_SUBDIR=quality/ to quality-eval - benchmark-tmpl.yml: quality-eval inputs, env vars, SCENARIO_SUBDIR routing, expanded upload patterns, --smoke in verify step - run-sweep.yml: sweep-quality-evals fan-out job - validation.py: 8 new Fields enum values, quality-eval Pydantic models - generate_sweep_configs.py: quality-eval generation in both sweep funcs - process_changelog.py: quality_evals bucket + routing - thresholds.yaml: thresholds for all 8 benchmarks (default + glm5.2) - validate_scores.py: --smoke flag, leniency when checked==0 - nvidia-master.yaml: glm5.2-quality-eval entry - runners.yaml: cluster:quality-eval -> bench-client_01 - ci-priority.yaml: quality-eval: 0.0 - Task configs: HLE YAMLs + utils.py, SWE-bench Pro config + runner
- nvidia-master.yaml: GPQA endpoint -> https://tokenplan.api.greennode.ai/v1 with smoke: true - benchmark-tmpl.yml: wire GN_TOKEN_PLAN secret as QUALITY_API_KEY - run-sweep.yml: eval-limit for sweep-enabled -> 10 (was 4) - perf-changelog.yaml: add glm5.2-quality-eval / quality-gpqa entry
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
The self-hosted runner on this machine is registered as bench-client_00, not bench-client_01.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33747490447 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33748366095 |
…-name routing
Quality-eval jobs can be picked up by any runner with the cluster:quality-eval
label (e.g. h200-greennode_03), not just bench-client_00. The existing
launch_${RUNNER_NAME%%_*}.sh routing assumes a matching launch script exists
for every runner name prefix, which fails with exit code 127 when a runner
like h200-greennode_03 picks up the job.
Route quality-eval scenarios (SCENARIO_SUBDIR=quality/) directly to
launch_quality-eval.sh, independent of the runner name.
质量评估任务可被任何带有 cluster:quality-eval 标签的 runner 领取（如
h200-greennode_03），而非仅 bench-client_00。现有基于 runner 名称前缀的
launch 脚本路由在无匹配脚本时以退出码 127 失败。将质量评估场景直接路由至
launch_quality-eval.sh，不再依赖 runner 名称。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33772346435 |
lm-eval's openai-chat-completions model requires the 'tenacity' package which is only installed via the 'api' extra (lm-eval[api]). Without it, the GPQA run fails with ModuleNotFoundError. Also bust the venv cache if tenacity is missing, so existing broken venvs are automatically repaired. lm-eval 的 openai-chat-completions 模型需要 'tenacity' 包，该包仅通过 'api' extra 安装。缺少时 GPQA 运行以 ModuleNotFoundError 失败。同时 在缺少 tenacity 时重建 venv 以自动修复。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33772767523 |
Add smoke run configurations for MMLU-Pro (20 Qs, ccu4), HLE (8 Qs, ccu4), LiveCodeBench (4 Qs, ccu4), BFCL (4 Qs, ccu1), SciCode (2 Qs, ccu2), SWE-bench Pro (1 task), and DeepSWE (1 task) against GreenNode endpoint. Introduce num-concurrent and eval-limit fields to the quality-eval config schema so per-benchmark API concurrency and sample counts can be specified in the search-space. Wire NUM_CONCURRENT through launch_quality-eval.sh to each benchmark script's concurrency env var (MULTIPROCESS, NUM_THREADS, MAX_CONNECTIONS, WORKERS, CCU). 为其余 7 个基准添加冒烟运行配置：MMLU-Pro（20 题, ccu4）、HLE（8 题, ccu4）、LiveCodeBench（4 题, ccu4）、BFCL（4 题, ccu1）、SciCode（2 题, ccu2）、SWE-bench Pro（1 任务）、DeepSWE（1 任务），使用 GreenNode 端点。 在 quality-eval 配置模式中引入 num-concurrent 和 eval-limit 字段，以便在 search-space 中指定每个基准的 API 并发数和样本数。
deepswe: correct repo URL datacurve/deep-swe → datacurve-ai/deep-swe scicode: replace invalid --retry-on-error flag with --max-retries swebench: correct PyPI package minisweagent → mini-swe-agent hle: add Pillow to lm-eval venv (HLE dataset has image features) livecodebench: pin anthropic<0.40 (HUMAN_PROMPT removed in newer SDK) bfcl: add soundfile to BFCL venv (qwen_agent dependency) 修复 6 个基准冒烟设置的错误： deepswe 仓库地址 datacurve/deep-swe → datacurve-ai/deep-swe scicode 无效参数 --retry-on-error → --max-retries swebench PyPI 包名 minisweagent → mini-swe-agent hle lm-eval 虚拟环境缺少 Pillow lcb anthropic SDK 版本过新导致 HUMAN_PROMPT 导入失败 bfcl 虚拟环境缺少 soundfile
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33774946695 |
…e flag
venv: add --clear to all 'uv venv' calls so cache-bust rebuilds
don't fail on 'virtual environment already exists'.
Cache is still reused when the bust-check passes (no rebuild).
deepswe: map EVAL_LIMIT -> N_TASKS (was hardcoded to 6, ignoring
eval-limit: 1 from config); add num-concurrent: 1 to config.
swebench: add num-concurrent: 1 to config for consistency.
scicode: drop invalid --continue-on-fail flag (not in this inspect-ai
version); --no-fail-on-error already covers it.
修复虚拟环境与冒烟参数问题：
venv 重建时加 --clear，避免 'venv already exists' 报错；
缓存健康时仍跳过重建，直接复用。
deepswe 将 EVAL_LIMIT 映射到 N_TASKS（原先硬编码 6，忽略配置）；
配置补充 num-concurrent: 1。
scicode 移除不支持的 --continue-on-fail 参数。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33783802385 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33785845762 |
swebench_pro: cache-bust check now tests 'import yaml' (pyyaml was
missing from stale cached venv).
livecodebench: drop anthropic<0.40 pin (LCB needs >=0.42.0); fix
cache-bust check to test 'import livecodebench'.
scicode: drop --metadata flag (unsupported on runner's inspect-ai).
bfcl: inject z-ai/glm-5.2-FC/-PROMPT into model_config.py after
clone (upstream BFCL doesn't have this model registered).
deepswe: collect_results now picks up result.json (singular) and
copies as results.json for benchmark-tmpl.yml's glob check.
job-naming: add benchmark-name to job display name so each quality
eval job is distinguishable in the CI dashboard.
config: comment out gpqa + mmlu_pro (smoke already passed in
run 33785845762); only dispatch the 6 remaining benchmarks.
…en_toks
swebench_pro: fix import path run.extra→run.benchmarks (mini-swe-agent
2.4.6 moved swebench module to run.benchmarks).
scicode: pin pyarrow<15 (PyExtensionType removed in pyarrow 15+,
cached venv had pyarrow 25.0.1); add 'import scicode'
cache-bust check.
hle: reduce max_gen_toks 32768→4096 in run_hle.sh + both task
yamls (HLE answers are ~400 tokens; 32K caused GreenNode
to allocate excessive resources and stall).
config: comment out deepswe (smoke passed in run 33832017880);
only dispatch 5 remaining: hle, livecodebench, bfcl,
scicode, swebench_pro.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33832017880 |
BFCL CLI has no --limit flag. When LIMIT is set, generate test_case_ids_to_generate.json with first N IDs per requested category and pass --run-ids. Without this, EVAL_LIMIT=4 was ignored and BFCL ran full eval (hundreds of test cases).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33835808262 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33845947257 |
…ults.json + scicode pyarrow
swebench_pro: clone with --recurse-submodules (SWE-agent is a git
submodule); init submodule if cache missed it; generate
instances.yaml via helper_code/generate_sweagent_instances.py
if missing (file is not in repo, must be generated from
HuggingFace dataset).
livecodebench: pin anthropic<0.40 (anthropic>=0.42 removed HUMAN_PROMPT
which lcb_runner still imports); cache-bust check now
verifies 'from anthropic import HUMAN_PROMPT'.
bfcl: write results.json wrapper from CSV scores so
benchmark-tmpl.yml's 'ls results*.json' check passes
(BFCL only produces CSVs, not JSONs).
scicode: install pyarrow<15 + datasets==2.14.4 BEFORE scicode
package (datasets pulls pyarrow>=15 which removes
PyExtensionType); cache-bust check now verifies
pyarrow.PyExtensionType exists.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33848804068 |
…aw_sample path
livecodebench: instead of pinning anthropic<0.40 (conflicts with
livecodebench pyproject anthropic>=0.42.0), patch
self_repair.py and test_output_prediction.py to add
try/except ImportError fallback for HUMAN_PROMPT/AI_PROMPT
(code_generation.py already has it).
scicode: pin numpy<2 alongside pyarrow<15 + datasets==2.14.4.
pyarrow 14.0.2 is compiled against numpy 1.x; when scicode
pulls numpy>=2, pyarrow fails with 'AttributeError:
_ARRAY_API not found'.
swebench_pro: fix RAW_SAMPLE default path from
data/swebench_pro_raw_sample.jsonl (does not exist) to
helper_code/sweap_eval_full_v2.jsonl (731 instances,
shipped with the repo).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33850821470 |
…name
livecodebench: LCB main.py has no --limit flag (uses --n for sample
count). Remove LIMIT handling entirely.
scicode: inspect-ai requires datasets>=2.16, but SciCode pyproject
pins unpinned 'datasets' → resolver picks 2.14.4.
Fix: install datasets>=2.16 + openai>=3.1 + anthropic +
config + other deps first, then scicode with --no-deps.
swebench_pro: OUT_DIR used 'swebench-pro' (hyphen) but collect_results
looks in 'swebench_pro' (underscore). Align to
swebench_pro so result files are found.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33855439087 |
LCB has a hardcoded LanguageModelStore dict and z-ai/glm-5.2 is not in it, causing KeyError. Patch lm_styles.py to add z-ai/glm-5.2 as an OpenAIChat model (uses OpenAIRunner which respects OPENAI_BASE_URL env var for the endpoint).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33856725913 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33857919213 |
…or swebench+lcb - LCB: patch code_generation.py to pass release_version as config name to load_dataset (cache has 'release_latest-version_tag=release_latest', not 'default-version_tag=release_latest') - SciCode: pin datasets==5.0.1 + pyarrow==25.0.1 (2.16.1 has TypeError bug with SciCode1/SciCode dataset in generate_from_dict) - SWE-bench Pro + LCB: copy eval_results.json / first .json as results.json so benchmark-tmpl.yml's 'ls results*.json' check passes LCB: 修复 load_dataset config name SciCode: 固定 datasets==5.0.1 SWE-bench Pro + LCB: 复制 results.json
…ection - LCB: patch main.py to read LCB_LIMIT env var and slice benchmark[:N] (LCB CLI has no --limit flag; without this it runs all 1055 problems) - SciCode: collect inspect-ai log files (.eval/.json) from logs/ subdir as results.json so benchmark-tmpl.yml's glob matches LCB: 添加 LCB_LIMIT 环境变量支持 SciCode: 收集 inspect-ai 日志文件
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33862609054 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33868837799 |
Patch all benchmark harnesses to use stream=True when calling the LLM API, avoiding proxy timeouts on long generations (32K tokens SciCode, 8K LCB, etc.). Each harness now accumulates SSE chunks into a response object matching the non-streaming format. - lm-eval (GPQA/MMLU-Pro/HLE): add stream=True to payload, parse SSE in model_call/amodel_call via _parse_sse_stream helpers - LiveCodeBench: stream=True in oai_runner.py, accumulate delta.content - BFCL: stream=True in generate_with_backoff, _accumulate_stream helper - SciCode/inspect_ai: async stream iteration in openai.py provider - SWE-bench Pro + DeepSWE: stream=True in litellm.completion, _accumulate_litellm_stream helper 为所有 8 个质量基准测试启用流式传输
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33880957352 |
1 participant
Smoke run GPQA-Diamond with 10 samples against GreenNode endpoint for GLM-5.2 quality validation.