From 9c9f1ff1fd97173dbed8b13b8aa8ecbad6b758f1 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Fri, 6 Nov 2026 02:36:37 -0500 Subject: [PATCH 001/111] =?UTF-8?q?feat(bench):=20add=20GLM-5.3-Flash=20FP?= =?UTF-8?q?8=20and=20NVFP4=20AgentX=20on=20GB300=20/=20feat(bench):=20?= =?UTF-8?q?=E6=96=B0=E5=A2=9E=20GB300=20=E4=B8=8A=E7=9A=84=20GLM-5.3-Flash?= =?UTF-8?q?=20FP8=20=E4=B8=8E=20NVFP4=20AgentX=20(#4715)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 GB300 AgentX lane * chore(bench): register GLM-5.3-Flash FP8 GB300 sweep * fix(ci): preserve canary gates after skipped reuse checks * fix(bench): use canonical Flash tokenizer ID for AgentX * feat(bench): add GLM-5.3-Flash FP8 and NVFP4 AgentX on GB300 Single-node SGLang v0.5.22 TP4 MTP 5/1/6 lanes for FP8 and NVFP4 using the SGLang cookbook GB300 recipe: GPU-resident KV at low concurrency and an auto-sized HiCache host tier (host memory fraction 0.90) at high concurrency. * feat(bench): move GLM-5.3-Flash GB300 lanes to SGLang v0.5.23 Pin lmsysorg/sglang:v0.5.23-cu130 for the FP8 and NVFP4 GB300 lanes and declare the expected SGLang version and FP8 checkpoint revision in the recipe identities. * revert GLM-5.3-Flash GB300 lanes to SGLang v0.5.22 Return both GB300 lanes to lmsysorg/sglang:v0.5.22-cu130 and declare SGLang 0.5.22 in the recipe identities. * fix(bench): record measured GB300 HiCache DRAM for GLM-5.3-Flash Set dram-utilization to 0.7495 (723 GB per node) on both GB300 lanes, covering the 708.7-722.9 GB per node the auto-sized HiCache pools allocated in run 42026173061. --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> --- .../sglang/gb300-fp4-mtp/agentic.yaml | 73 +++++++++++++++++++ .../sglang/gb300-fp8-mtp/agentic.yaml | 72 ++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 35 +++++++++ inferencex-e2e/configs/runners.yaml | 2 + inferencex-e2e/perf-changelog.yaml | 11 +++ 5 files changed, 193 insertions(+) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp4-mtp/agentic.yaml create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp8-mtp/agentic.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp4-mtp/agentic.yaml new file mode 100644 index 0000000000..b27a420d2f --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp4-mtp/agentic.yaml @@ -0,0 +1,73 @@ +base: + schema: 2 + name: glm5.3flash-fp4-gb300-sglang-agentic-mtp + identity: + model: + revision: 1d00738646cbb0c6d0a9eb403dac5f366f1473a7 + container: {} + frameworks: + sglang: 0.5.22 + resources: + gpu_type: gb300 + gpus_per_node: 4 + slurm: + time_limit: '4:00:00' + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 4 + args: + served-model-name: nvidia/GLM-5.3-Flash-NVFP4 + tensor-parallel-size: 4 + expert-parallel-size: 1 + quantization: modelopt_fp4 + context-length: 1048576 + dsa-prefill-backend: trtllm + dsa-decode-backend: trtllm + kv-cache-dtype: fp8_e4m3 + moe-runner-backend: flashinfer_trtllm + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + reasoning-parser: auto + tool-call-parser: auto + watchdog-timeout: 1800 + enable-metrics: true + env: + PYTHONNOUSERSITE: '1' + PYTHONUNBUFFERED: '1' + HF_HUB_OFFLINE: '1' + SGLANG_TIMEOUT_KEEP_ALIVE: '900' + health_check: + max_attempts: 1440 + interval_seconds: 5 + benchmark: + env: + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_gpu_kv: + benchmark: + env: + KV_OFFLOADING: none + +override_hicache: + roles: + agg: + args: + enable-hierarchical-cache: true + # The 0.8 default sizes the host pool at about 1.0x the device KV pool on GB300. + hicache-host-memory-fraction: 0.90 + benchmark: + env: + KV_OFFLOADING: dram diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp8-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp8-mtp/agentic.yaml new file mode 100644 index 0000000000..17f27639d4 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/gb300-fp8-mtp/agentic.yaml @@ -0,0 +1,72 @@ +base: + schema: 2 + name: glm5.3flash-fp8-gb300-sglang-agentic-mtp + identity: + model: + revision: 46a050bec13002ad29ee795c16e61cf4fa3c4a9c + container: {} + frameworks: + sglang: 0.5.22 + resources: + gpu_type: gb300 + gpus_per_node: 4 + slurm: + time_limit: '4:00:00' + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 4 + args: + served-model-name: zai-org/GLM-5.3-Flash + tensor-parallel-size: 4 + expert-parallel-size: 1 + context-length: 1048576 + dsa-prefill-backend: trtllm + dsa-decode-backend: trtllm + kv-cache-dtype: fp8_e4m3 + moe-runner-backend: flashinfer_trtllm + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + reasoning-parser: auto + tool-call-parser: auto + watchdog-timeout: 1800 + enable-metrics: true + env: + PYTHONNOUSERSITE: '1' + PYTHONUNBUFFERED: '1' + HF_HUB_OFFLINE: '1' + SGLANG_TIMEOUT_KEEP_ALIVE: '900' + health_check: + max_attempts: 1440 + interval_seconds: 5 + benchmark: + env: + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_gpu_kv: + benchmark: + env: + KV_OFFLOADING: none + +override_hicache: + roles: + agg: + args: + enable-hierarchical-cache: true + # The 0.8 default sizes the host pool at about 1.0x the device KV pool on GB300. + hicache-host-memory-fraction: 0.90 + benchmark: + env: + KV_OFFLOADING: dram diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 316ca2b225..25b6d70b42 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5729,6 +5729,41 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp: tp: 8 ep: 8 dp-attn: true + +glm5.3flash-fp8-gb300-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + model: zai-org/GLM-5.3-Flash + model-prefix: glm5.3flash + runner: cluster:gb300-nv + precision: fp8 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/gb300-fp8-mtp + scenarios: + agentic-coding: + # HiCache auto-sizes at fraction 0.90; run 42026173061 measured 708.7-722.9 GB per node (4 ranks) of 964.7 GB. + - dram-utilization: 0.7495 + search-space: + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32], srt-recipe: "agentic.yaml:override_gpu_kv" } + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [40, 44, 48], srt-recipe: "agentic.yaml:override_hicache" } + +glm5.3flash-fp4-gb300-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + model: nvidia/GLM-5.3-Flash-NVFP4 + model-prefix: glm5.3flash + runner: cluster:gb300-nv + precision: fp4 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/gb300-fp4-mtp + scenarios: + agentic-coding: + # HiCache auto-sizes at fraction 0.90; run 42026173061 measured 708.7-722.9 GB per node (4 ranks) of 964.7 GB. + - dram-utilization: 0.7495 + search-space: + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32], srt-recipe: "agentic.yaml:override_gpu_kv" } + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [40, 44, 48], srt-recipe: "agentic.yaml:override_hicache" } + # H100 AgentX arm for DeepSeek-V4.1-Flash. H100 is not in the upstream hardware # table (h200, gb200, gb300, mi350x are). It needs its own script, not the # shared symlink: the sparse attention indexer's diff --git a/inferencex-e2e/configs/runners.yaml b/inferencex-e2e/configs/runners.yaml index a48a0bd976..4b0e615d8f 100644 --- a/inferencex-e2e/configs/runners.yaml +++ b/inferencex-e2e/configs/runners.yaml @@ -639,6 +639,8 @@ clusters: DeepSeek-R1-0528: {root: scratch, dir: DeepSeek-R1-0528} DeepSeek-R1-0528-NVFP4-v2: {root: scratch, dir: DeepSeek-R1-0528-NVFP4-v2} GLM-5.2-NVFP4: {root: scratch, dir: GLM-5.2-NVFP4} + GLM-5.3-Flash: {root: data-models, dir: GLM-5.3-Flash} + GLM-5.3-Flash-NVFP4: {root: data-models, dir: GLM-5.3-Flash-NVFP4} Kimi-K3.1-Pro: {root: scratch, dir: Kimi-K3.1-Pro} MiniMax-M3.1-Flash: {root: data-models, dir: MiniMax-M3.1-Flash} Muse-Spark-Open-816B-A42B-NVFP4: {root: data-models, dir: Muse-Spark-Open-816B-A42B-NVFP4} diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 81d3260a6c..ba2465a656 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12224,3 +12224,14 @@ - "GPU-resident KV at concurrency 2-16; HiCache with SGLang's default auto-sized host tier (~119 GiB, 127 GB per rank; ~1,020 GB per node) at concurrency 20, 24, 28 and 32." - "Prefill CUDA graphs are disabled after a CUDA misaligned-address fault during ~336k-token single-request replay at concurrency 1." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4719 + +- config-keys: + - glm5.3flash-fp8-gb300-sglang-agentic-mtp + - glm5.3flash-fp4-gb300-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Add native SGLang GLM-5.3-Flash AgentX on one GB300 tray (TP4/EP1) with lmsysorg/sglang:v0.5.22-cu130, FP8 (zai-org/GLM-5.3-Flash) and NVFP4 (nvidia/GLM-5.3-Flash-NVFP4), embedded EAGLE MTP 5/1/6." + - "SGLang cookbook GB300 recipe: TRT-LLM DSA prefill and decode, FP8 E4M3 KV cache, flashinfer_trtllm MoE, auto reasoning and tool-call parsers; NVFP4 adds modelopt_fp4 quantization." + - "GPU-resident KV at concurrency 1, 4, 8, 16, 24 and 32; auto-sized HiCache host tier (--hicache-host-memory-fraction 0.90, recorded as 723 GB per node, the measured maximum) at concurrency 40, 44 and 48." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4715 From 23a74c6acf1de0c2b4196288f1217eff14361fb1 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Fri, 6 Nov 2026 02:40:00 -0500 Subject: [PATCH 002/111] =?UTF-8?q?Add=20GLM-5.3-Flash=20FP8=20and=20NVFP4?= =?UTF-8?q?=20B300=20SGLang=20AgentX=20/=20=E6=B7=BB=E5=8A=A0=20GLM-5.3-Fl?= =?UTF-8?q?ash=20FP8=20=E4=B8=8E=20NVFP4=20B300=20SGLang=20AgentX=20(#4710?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * feat: add GLM-5.3-Flash FP8 B300 AgentX recipe * perf: register GLM-5.3-Flash FP8 B300 sweep * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * fix: use SGLang v0.5.22 decode CUDA graph flag * feat(agentx): add GLM-5.3-Flash FP8 and NVFP4 B300 SGLang MTP lanes Serve both precisions with the SGLang cookbook's B300 Low Latency flags: GPU-resident KV at concurrency up to 48, and at concurrency 64 and 96 HiCache with auto-sized host pools (fraction 0.76, within the 3 TB node cap) and max-running-requests 96, which SGLang otherwise resets to 48 under EAGLE. * fix(agentx): record measured HiCache DRAM for GLM-5.3-Flash B300 Run 42024862281 allocated up to 2802.2 GB (FP8) and 2813.6 GB (NVFP4) of HiCache host pools per node; dram-utilization 0.935 and 0.939 record 2804 and 2816 GB. --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> --- .../sglang/b300-fp4-mtp/agentic.yaml | 73 +++++++++++++++++++ .../sglang/b300-fp8-mtp/agentic.yaml | 69 ++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 32 ++++++++ inferencex-e2e/configs/runners.yaml | 2 + inferencex-e2e/perf-changelog.yaml | 10 +++ 5 files changed, 186 insertions(+) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp4-mtp/agentic.yaml create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp4-mtp/agentic.yaml new file mode 100644 index 0000000000..d76a5fde3f --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp4-mtp/agentic.yaml @@ -0,0 +1,73 @@ +base: + schema: 2 + name: glm5.3flash-fp4-b300-sglang-agentic-mtp + identity: + model: + revision: 1d00738646cbb0c6d0a9eb403dac5f366f1473a7 + resources: + gpu_type: b300 + gpus_per_node: 8 + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 8 + # Serve flags follow the cookbook's B300 Low Latency cell at + # https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash. + args: + served-model-name: nvidia/GLM-5.3-Flash-NVFP4 + quantization: modelopt_fp4 + tensor-parallel-size: 8 + dsa-prefill-backend: trtllm + dsa-decode-backend: trtllm + kv-cache-dtype: fp8_e4m3 + moe-runner-backend: flashinfer_trtllm + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + reasoning-parser: auto + tool-call-parser: auto + watchdog-timeout: 1800 + enable-metrics: true + env: + PYTHONNOUSERSITE: '1' + # Both default off in the pinned image; explicit so the shipped draft head keeps its stored precision. + SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '0' + SGLANG_GLM_NEXTN_MOE_PTPC: '0' + # Outlast AIPerf's pooled connections past Uvicorn's keep-alive. + SGLANG_TIMEOUT_KEEP_ALIVE: '1800' + # Keep FlashInfer, Triton and JIT caches allocation-local. + SGLANG_CACHE_DIR: /tmp/sglang-cache + benchmark: + env: + # Warmup can leave bytes unacknowledged past AIPerf's 30 s default. + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_none: + benchmark: + env: + KV_OFFLOADING: none + +override_dram: + roles: + agg: + args: + enable-hierarchical-cache: true + # Auto-sized host pools, not the cookbook's 32 GB/rank tier, which is smaller than the + # device pool. 0.76 is the largest fraction whose pools fit the 3 TB node DRAM cap. + hicache-host-memory-fraction: 0.76 + # SGLang resets admission to 48 under EAGLE when unset; 96 is the measured-safe cap. + max-running-requests: 96 + benchmark: + env: + KV_OFFLOADING: dram diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml new file mode 100644 index 0000000000..19db169d69 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml @@ -0,0 +1,69 @@ +base: + schema: 2 + name: glm5.3flash-fp8-b300-sglang-agentic-mtp + resources: + gpu_type: b300 + gpus_per_node: 8 + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 8 + # Serve flags follow the cookbook's B300 Low Latency cell at + # https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash. + args: + served-model-name: zai-org/GLM-5.3-Flash + tensor-parallel-size: 8 + dsa-prefill-backend: trtllm + dsa-decode-backend: trtllm + kv-cache-dtype: fp8_e4m3 + moe-runner-backend: flashinfer_trtllm + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + reasoning-parser: auto + tool-call-parser: auto + watchdog-timeout: 1800 + enable-metrics: true + env: + PYTHONNOUSERSITE: '1' + # Both default off in the pinned image; explicit so the shipped draft head keeps its stored precision. + SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '0' + SGLANG_GLM_NEXTN_MOE_PTPC: '0' + # Outlast AIPerf's pooled connections past Uvicorn's keep-alive. + SGLANG_TIMEOUT_KEEP_ALIVE: '1800' + # Keep FlashInfer, Triton and JIT caches allocation-local. + SGLANG_CACHE_DIR: /tmp/sglang-cache + benchmark: + env: + # Warmup can leave bytes unacknowledged past AIPerf's 30 s default. + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_none: + benchmark: + env: + KV_OFFLOADING: none + +override_dram: + roles: + agg: + args: + enable-hierarchical-cache: true + # Auto-sized host pools, not the cookbook's 32 GB/rank tier, which is smaller than the + # device pool. 0.76 is the largest fraction whose pools fit the 3 TB node DRAM cap. + hicache-host-memory-fraction: 0.76 + # SGLang resets admission to 48 under EAGLE when unset; 96 is the measured-safe cap. + max-running-requests: 96 + benchmark: + env: + KV_OFFLOADING: dram diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 25b6d70b42..269bac4c97 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7499,6 +7499,38 @@ glm5.3-fp4-b300-sglang-agentic-mtp: search-space: - { tp: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [22, 24, 25, 28], srt-recipe: agentic.yaml } +glm5.3flash-fp8-b300-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + model: zai-org/GLM-5.3-Flash + model-prefix: glm5.3flash + runner: cluster:b300-dsxe + precision: fp8 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/b300-fp8-mtp + scenarios: + agentic-coding: + - dram-utilization: 0.935 + search-space: + - { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32, 48], srt-recipe: "agentic.yaml:override_none" } + - { tp: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64, 96], srt-recipe: "agentic.yaml:override_dram" } + +glm5.3flash-fp4-b300-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + model: nvidia/GLM-5.3-Flash-NVFP4 + model-prefix: glm5.3flash + runner: cluster:b300-dsxe + precision: fp4 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/b300-fp4-mtp + scenarios: + agentic-coding: + - dram-utilization: 0.939 + search-space: + - { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [4, 8, 16, 24, 32, 48], srt-recipe: "agentic.yaml:override_none" } + - { tp: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64, 96], srt-recipe: "agentic.yaml:override_dram" } + minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-agg: image: vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d76347628973fb80519a919a2f2740b6645b62cba3fe2bd87b25f111bd0e948 model: MiniMaxAI/MiniMax-M3.1-Flash diff --git a/inferencex-e2e/configs/runners.yaml b/inferencex-e2e/configs/runners.yaml index 4b0e615d8f..fd14da3302 100644 --- a/inferencex-e2e/configs/runners.yaml +++ b/inferencex-e2e/configs/runners.yaml @@ -525,6 +525,8 @@ clusters: GLM-5.2-FP8: {root: scratch, dir: GLM-5.2-FP8} GLM-5.2-NVFP4: {root: scratch, dir: GLM-5.2-NVFP4} GLM-5.3-NVFP4: {root: scratch, dir: GLM-5.3-NVFP4} + GLM-5.3-Flash: {root: data-models, dir: GLM-5.3-Flash} + GLM-5.3-Flash-NVFP4: {root: data-models, dir: GLM-5.3-Flash-NVFP4} Kimi-K3: {root: scratch, dir: Kimi-K3} Kimi-K3.1-Pro: {root: scratch, dir: Kimi-K3.1-Pro} MiniMax-M3: {root: scratch, dir: MiniMax-M3} diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index ba2465a656..af423ffccd 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12235,3 +12235,13 @@ - "SGLang cookbook GB300 recipe: TRT-LLM DSA prefill and decode, FP8 E4M3 KV cache, flashinfer_trtllm MoE, auto reasoning and tool-call parsers; NVFP4 adds modelopt_fp4 quantization." - "GPU-resident KV at concurrency 1, 4, 8, 16, 24 and 32; auto-sized HiCache host tier (--hicache-host-memory-fraction 0.90, recorded as 723 GB per node, the measured maximum) at concurrency 40, 44 and 48." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4715 + +- config-keys: + - glm5.3flash-fp8-b300-sglang-agentic-mtp + - glm5.3flash-fp4-b300-sglang-agentic-mtp + scenario-type: [agentic-coding] + description: + - "Add GLM-5.3-Flash FP8 (zai-org/GLM-5.3-Flash) and NVFP4 (nvidia/GLM-5.3-Flash-NVFP4) AgentX on one 8x B300 node with the digest-pinned official SGLang v0.5.22-cu130 image: TP8, EAGLE MTP 5/1/6, TRT-LLM DSA, FP8 KV and FlashInfer TRT-LLM MoE, as in the SGLang cookbook's B300 cells." + - "GPU-resident KV at concurrency 1/4/8/16/24/32/48 (FP8) and 4/8/16/24/32/48 (NVFP4); HiCache at concurrency 64/96 with auto-sized host pools (host-memory fraction 0.76, about 2.8 TB per node, within the 3 TB cap) instead of the cookbook's 32 GB per rank, which is smaller than the device pool, and max-running-requests 96, since SGLang otherwise resets admission to 48 under EAGLE. Engine defaults set the remaining memory, admission and CUDA graph sizes." + - "Acceptance comes from the glm5.3flash MTP golden curve; the shipped draft head is served as stored, with SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE off." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4710 From b4697c9fdec29b464f8f60a5a834f89b79fc80aa Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Fri, 6 Nov 2026 02:44:19 -0500 Subject: [PATCH 003/111] =?UTF-8?q?Add=20GLM-5.3-Flash=20FP8=20H100=20SGLa?= =?UTF-8?q?ng=20AgentX=20with=20HiCache=20/=20=E6=96=B0=E5=A2=9E=20GLM-5.3?= =?UTF-8?q?-Flash=20FP8=20H100=20SGLang=20AgentX=EF=BC=88=E5=90=AB=20HiCac?= =?UTF-8?q?he=EF=BC=89=20(#4717)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 H100 AgentX lane * chore(bench): register H100 GLM-5.3-Flash full sweep * chore(debug): probe H100 checkpoint inventory read-only * fix(ci): preserve canary gates after skipped reuse checks * chore(debug): remove completed H100 inventory probe * fix(bench): stage pinned GLM-5.3-Flash checkpoint for H100 * fix(bench): verify pinned H100 cache before offline serving * fix(bench): pin H100 server tokenizer to cached checkpoint * fix(bench): retain canonical AgentX tokenizer resolution on H100 * fix(bench): clear offline flags for the H100 AgentX client * fix(bench): test eager prefill compatibility on H100 * feat(bench): consolidate GLM-5.3-Flash FP8 H100 AgentX onto the cookbook recipe Rebuild the H100 lane on main with the SGLang cookbook H100 low-latency FP8 recipe: GPU-resident KV at concurrency 1-2 and the cookbook HiCache L1+L2 host tier (32 GB of host KV per rank) at concurrency 4-16. Keep prefill CUDA graphs disabled for the H100 long-context replay fault and stage the pinned checkpoint into the shared HF cache. * fix(bench): record the H100 HiCache host allocation as its DRAM share * fix(bench): drop the no-op trust-remote-code from the H100 recipe * fix(bench): size H100 HiCache DRAM from the KV/Mamba split and move points to c4-c12 --hicache-size 32 is shared by the KV and Mamba host pools in proportion to their device bytes, so host KV is about 22 GB per rank; with the DSA indexer pool each rank allocates about 32.7 GB. * fix(bench): drop H100 HiCache c12 and restate the HiCache host allocation * chore(bench): drop H100 GLM-5.3-Flash checkpoint staging The pinned checkpoint is already in the shared H100 Hugging Face cache, which the server reads offline through its pinned tokenizer path, as the MI300X and MI325X lanes on main do. * chore: refresh PR #4717 for sweep reuse [skip-sweep] --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> --- .../sglang/h100-fp8-mtp/agentic.yaml | 75 +++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 18 +++++ inferencex-e2e/perf-changelog.yaml | 10 +++ 3 files changed, 103 insertions(+) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/h100-fp8-mtp/agentic.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/h100-fp8-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/h100-fp8-mtp/agentic.yaml new file mode 100644 index 0000000000..73174e49d6 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/h100-fp8-mtp/agentic.yaml @@ -0,0 +1,75 @@ +base: + schema: 2 + name: glm5.3flash-fp8-h100-sglang-agentic-mtp + identity: + model: + revision: 46a050bec13002ad29ee795c16e61cf4fa3c4a9c + resources: + gpu_type: h100 + gpus_per_node: 8 + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 8 + args: + served-model-name: zai-org/GLM-5.3-Flash + revision: 46a050bec13002ad29ee795c16e61cf4fa3c4a9c + tokenizer-path: /mnt/hf_hub_cache/models--zai-org--GLM-5.3-Flash/snapshots/46a050bec13002ad29ee795c16e61cf4fa3c4a9c + tensor-parallel-size: 8 + expert-parallel-size: 8 + mem-fraction-static: 0.70 + dsa-prefill-backend: tilelang + dsa-decode-backend: tilelang + kv-cache-dtype: bfloat16 + moe-runner-backend: deep_gemm + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + reasoning-parser: auto + tool-call-parser: auto + # v0.5.22's default prefill CUDA graph faults on H100 long-context replay (illegal address). + cuda-graph-backend-prefill: disabled + watchdog-timeout: 1800 + enable-cache-report: true + enable-metrics: true + env: + HF_HUB_OFFLINE: '1' + TRANSFORMERS_OFFLINE: '1' + PYTHONNOUSERSITE: '1' + # Outlast AIPerf's pooled connections past Uvicorn's keep-alive. + SGLANG_TIMEOUT_KEEP_ALIVE: '900' + health_check: + max_attempts: 720 + interval_seconds: 10 + benchmark: + env: + HF_HUB_OFFLINE: '' + TRANSFORMERS_OFFLINE: '' + # Warmup can leave bytes unacknowledged past AIPerf's 30 s default. + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_none: + benchmark: + env: + KV_OFFLOADING: none + +override_hicache: + roles: + agg: + args: + enable-hierarchical-cache: true + hicache-size: 32 + benchmark: + env: + KV_OFFLOADING: dram diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 269bac4c97..d486c9f442 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7376,6 +7376,24 @@ musespark-fp4-h100-sglang-agentic-dflash: search-space: - { tp: 8, ep: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [1, 2, 4, 8], srt-recipe: agentic.yaml } +glm5.3flash-fp8-h100-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + model: zai-org/GLM-5.3-Flash + model-prefix: glm5.3flash + runner: cluster:h100-dsxe + precision: fp8 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/h100-fp8-mtp + scenarios: + agentic-coding: + - search-space: + - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 2], srt-recipe: 'agentic.yaml:override_none' } + # HiCache allocates ~32.5 GB/rank, ~260 GB/node: 32 GB split over KV and Mamba host pools plus the DSA indexer pool. + - dram-utilization: 0.131 + search-space: + - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [4, 6, 8, 10], srt-recipe: 'agentic.yaml:override_hicache' } + glm5.3flash-fp8-b200-sglang-agentic-mtp: image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae model: zai-org/GLM-5.3-Flash diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index af423ffccd..18b0f5d5a6 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12245,3 +12245,13 @@ - "GPU-resident KV at concurrency 1/4/8/16/24/32/48 (FP8) and 4/8/16/24/32/48 (NVFP4); HiCache at concurrency 64/96 with auto-sized host pools (host-memory fraction 0.76, about 2.8 TB per node, within the 3 TB cap) instead of the cookbook's 32 GB per rank, which is smaller than the device pool, and max-running-requests 96, since SGLang otherwise resets admission to 48 under EAGLE. Engine defaults set the remaining memory, admission and CUDA graph sizes." - "Acceptance comes from the glm5.3flash MTP golden curve; the shipped draft head is served as stored, with SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE off." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4710 + +- config-keys: + - glm5.3flash-fp8-h100-sglang-agentic-mtp + scenario-type: [agentic-coding] + description: + - "Add native GLM-5.3-Flash FP8 AgentX on one 8x H100 node with the digest-pinned official SGLang v0.5.22-cu130 image and the SGLang cookbook H100 low-latency recipe: TP8/EP8, BF16 KV, TileLang DSA, DeepGEMM MoE, mem-fraction-static 0.70, and the embedded MTP head as EAGLE 5/1/6." + - "GPU-resident KV at concurrency 1 and 2; cookbook HiCache L1+L2 host tier (--hicache-size 32 per rank) at concurrency 4, 6, 8 and 10." + - "Prefill CUDA graphs are disabled because the image's default prefill graph faults with an illegal address during long-context replay on H100; decode CUDA graphs are unchanged." + - "Serve the pinned checkpoint revision 46a050bec13002ad29ee795c16e61cf4fa3c4a9c offline from the shared H100 Hugging Face cache." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4717 From af33fcba5454e8d343e631de5e5f0dfbf59ecf5b Mon Sep 17 00:00:00 2001 From: Jiangyun Zhu Date: Fri, 6 Nov 2026 16:57:25 +0800 Subject: [PATCH 004/111] [K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths (#4833) Run the Kimi-K3.1-Pro B300 vLLM recipe's low-latency decode (decode_backend=latency_specialized) at c1-c4, run c1-c2 at plain TP8 instead of TP8 x DCP8, match the SGLang recipe's DSpark draft depths, size each decode graph cap to max-num-seqs x (1 + drafts), and raise the engine-ready timeout and health check to 7200 s. Co-authored-by: zjy0516 --- .../kimik3.1/vllm/b300-fp4-mtp/agentic.yaml | 97 +++++++++++-------- inferencex-e2e/configs/nvidia-master.yaml | 5 +- inferencex-e2e/perf-changelog.yaml | 7 ++ 3 files changed, 67 insertions(+), 42 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/b300-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/b300-fp4-mtp/agentic.yaml index a6ad375bbd..21413562a5 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/b300-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/b300-fp4-mtp/agentic.yaml @@ -14,10 +14,11 @@ base: engine: type: vllm connector: null - # ~1.56 TB of MXFP4 shards may take the full VLLM_ENGINE_READY_TIMEOUT_S. + # Loading ~1.56 TB of MXFP4 shards plus the latency_specialized decode autotune can take + # most of VLLM_ENGINE_READY_TIMEOUT_S. health_check: interval_seconds: 10 - max_attempts: 360 + max_attempts: 720 roles: agg: nodes: 1 @@ -26,8 +27,6 @@ base: args: served-model-name: moonshotai/Kimi-K3.1-Pro tensor-parallel-size: 8 - # Shards the MLA latent and indexer-key caches across the TP group: ~7x the KV tokens of plain TP8. - decode-context-parallel-size: 8 trust-remote-code: true language-model-only: true load-format: fastsafetensors @@ -46,7 +45,7 @@ base: # TOTAL_CPU_DRAM_GB split across the 8 TP ranks. kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":281125000000,"lazy_offload":false}}' env: - VLLM_ENGINE_READY_TIMEOUT_S: '3600' + VLLM_ENGINE_READY_TIMEOUT_S: '7200' PYTHONUNBUFFERED: '1' # Identical prefixes must hash to identical block keys run-to-run. PYTHONHASHSEED: '42' @@ -58,125 +57,143 @@ base: KV_OFFLOADING: dram TOTAL_CPU_DRAM_GB: '2249' -# Admission is 2x CONC. The decode graphs cover max-num-seqs x (1 + draft tokens); -# throughput runs replace block rejection with the golden acceptance length. -override_c1: +# Admission is 2x CONC. DSpark drafts match the SGLang recipe's DFLASH drafts: 7 through c8, +# 6 at c16, 5 at c24-c40 and 4 from c48. The largest decode graph covers max-num-seqs x (1 + drafts), +# rounded up to a default capture size; throughput runs replace block rejection with the +# golden acceptance length. Plain TP8 at c1-c2, where the KV fits on GPU and DCP's decode +# collectives cost most; DCP8 from c4 for ~7x the KV tokens. +# Low-latency decode (the fused CuTe DSL path) runs only where every decode graph has at +# most 64 rows (c1-c4): this image launches its 65-320-row MoE experts on 152 CTAs that +# must all be resident, which a 148-SM B300 cannot hold. +override_tp8_c1: roles: agg: args: + additional-config: '{"decode_backend":"latency_specialized"}' max-num-seqs: 2 - max-cudagraph-capture-size: 12 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 16 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '1' -override_c2: +override_tp8_c2: roles: agg: args: + additional-config: '{"decode_backend":"latency_specialized"}' max-num-seqs: 4 - max-cudagraph-capture-size: 24 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 32 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '2' -override_c4: +override_dcp8_c4: roles: agg: args: + additional-config: '{"decode_backend":"latency_specialized"}' + decode-context-parallel-size: 8 max-num-seqs: 8 - max-cudagraph-capture-size: 48 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 64 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '4' -override_c8: +override_dcp8_c8: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 16 - max-cudagraph-capture-size: 96 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 128 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '8' -override_c16: +override_dcp8_c16: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 32 - max-cudagraph-capture-size: 160 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 224 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":6,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '16' -override_c24: +override_dcp8_c24: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 48 - max-cudagraph-capture-size: 240 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 288 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '24' -override_c32: +override_dcp8_c32: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 64 - max-cudagraph-capture-size: 256 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 384 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '32' -override_c40: +override_dcp8_c40: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 80 - max-cudagraph-capture-size: 320 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 480 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '40' -override_c48: +override_dcp8_c48: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 96 - max-cudagraph-capture-size: 384 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 480 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '48' -override_c56: +override_dcp8_c56: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 112 - max-cudagraph-capture-size: 336 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":2,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 560 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '56' -override_c70: +override_dcp8_c70: roles: agg: args: + decode-context-parallel-size: 8 max-num-seqs: 140 - max-cudagraph-capture-size: 420 - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":2,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + max-cudagraph-capture-size: 704 + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '70' diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index d486c9f442..521117a5f3 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5957,8 +5957,9 @@ kimik3.1-fp4-b300-vllm-agentic-dspark: # 0.75 resolves to 2249 GB of host DRAM, which the recipe's Simple CPU offload splits across the 8 TP ranks. - dram-utilization: 0.75 search-space: - # The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. DSpark drafts 5 tokens up to conc 8, then 4, 3 and 2. - - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 70], srt-recipe: agentic.yaml } + # The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. DSpark drafts 7 tokens up to conc 8, then 6, 5 and 4. + - { tp: 8, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [1, 2], srt-recipe: agentic.yaml } + - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 8, 16, 24, 32, 40, 48, 56, 70], srt-recipe: agentic.yaml } dsv41flash-fp4-b200-vllm-agentic-dspark: image: vllm/vllm-openai:cu134-nightly-40941f2c4ba96fba0a7e282a5355e89cd74d9ac5@sha256:13e92fcd97268013d13d07fb97d80803227cf0beae0161f8dce95a8200eaafb0 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 18b0f5d5a6..1db54f77c1 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12255,3 +12255,10 @@ - "Prefill CUDA graphs are disabled because the image's default prefill graph faults with an illegal address during long-context replay on H100; decode CUDA graphs are unchanged." - "Serve the pinned checkpoint revision 46a050bec13002ad29ee795c16e61cf4fa3c4a9c offline from the shared H100 Hugging Face cache." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4717 + +- config-keys: + - kimik3.1-fp4-b300-vllm-agentic-dspark + description: + - "Kimi-K3.1-Pro B300 vLLM: enable the recipe's low-latency decode (decode_backend=latency_specialized) at c1-c4, whose decode graphs stay within 64 rows (the image's wider MoE needs 152 resident CTAs, more than a B300's 148 SMs), and run c1-c2 at plain TP8 instead of TP8 x DCP8." + - "Match the SGLang recipe's draft depths (DSpark 7 tokens through c8, 6 at c16, 5 at c24-c40, 4 from c48), size each decode graph cap to max-num-seqs x (1 + drafts), and raise the engine-ready timeout and health check to 7200 s." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4833 From c872d1a332ed5bcc6ec5ee625a6ce5b98a1f5c75 Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Fri, 6 Nov 2026 01:48:02 -0800 Subject: [PATCH 005/111] =?UTF-8?q?[VideoGenX]=20sweep=20H3=20latency=20on?= =?UTF-8?q?=201,=202,=204=20and=208=20B200=20GPUs=20/=20[VideoGenX]=20?= =?UTF-8?q?=E5=9C=A8=201=E3=80=812=E3=80=814=E3=80=818=20=E5=BC=A0=20B200?= =?UTF-8?q?=20=E4=B8=8A=E6=89=AB=E6=8F=8F=20H3=20=E5=BB=B6=E8=BF=9F=20(#48?= =?UTF-8?q?52)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: sweep H3 video on 1, 2, 4 and 8 B200 GPUs * chore: link B200 GPU-count changelog entry to #4852 --- .../h3/sglang/b200-bf16/serving-client.yaml | 2 +- inferencex-e2e/configs/nvidia-master.yaml | 9 +++++++++ inferencex-e2e/docs/MODELS.md | 2 +- inferencex-e2e/docs/video-generation.md | 5 +++-- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 5 files changed, 21 insertions(+), 4 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/b200-bf16/serving-client.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/b200-bf16/serving-client.yaml index d3cbc3f117..83c575a2bb 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/b200-bf16/serving-client.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/b200-bf16/serving-client.yaml @@ -32,7 +32,7 @@ services: - --num-gpus - "{gpus_per_node}" - --ulysses-degree - - "4" + - "{gpus_per_node}" - --tp-size - "1" - --encoder-parallel diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 521117a5f3..40db965f44 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7497,9 +7497,18 @@ h3-bf16-b200-sglang-video: - spec: h3-v2.yaml set: release-v1 search-space: + - gpus: 1 + srt-recipe: serving-client.yaml + conc-list: [1] + - gpus: 2 + srt-recipe: serving-client.yaml + conc-list: [1] - gpus: 4 srt-recipe: serving-client.yaml conc-list: [1] + - gpus: 8 + srt-recipe: serving-client.yaml + conc-list: [1] glm5.3-fp4-b300-sglang-agentic-mtp: image: lmsysorg/sglang:v0.5.20-cu130@sha256:b27fce60bc5494c118c4910702812bcfa8cee67abcdd1ff8b0902f21647552f4 diff --git a/inferencex-e2e/docs/MODELS.md b/inferencex-e2e/docs/MODELS.md index f67e268347..404327641b 100644 --- a/inferencex-e2e/docs/MODELS.md +++ b/inferencex-e2e/docs/MODELS.md @@ -36,7 +36,7 @@ For non-grandfathered models, `ROCM_QUICK_REDUCE_QUANTIZATION`, `VLLM_ROCM_QUICK |---|---|---| | Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see the srt-slurm recipes under [`benchmarks/single_node/srt-slurm-recipes/`](../benchmarks/single_node/srt-slurm-recipes) and the shared client [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation policy](#deprecation-policy)). | | Single-turn 8k1k | 8192 / 1024 | Active. This is the primary fixed-sequence-length scenario. | -| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: H100/B200/MI355X SGLang select four-GPU, 20-clip `h3-v2.yaml` C1 points. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | +| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 GPUs and 4 H100 or MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | | Single-turn 1k1k | 1024 / 1024 | Deprecated since 2026-07-17 ([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Archived configs lived in `configs/deprecated/` until it was deleted (#3463); see git history. The GLM-5.1 B200 TileRT point added later in [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) was the last 1k1k config; it was retired on 2026-10-01 ([#3638](https://github.com/SemiAnalysisAI/InferenceX/pull/3638)). | | Single-turn 1k8k | 1024 / 8192 | **Deprecated for all models** since 2026-03-27 ([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Configs were removed, not archived. | diff --git a/inferencex-e2e/docs/video-generation.md b/inferencex-e2e/docs/video-generation.md index 2263251293..090962dcb5 100644 --- a/inferencex-e2e/docs/video-generation.md +++ b/inferencex-e2e/docs/video-generation.md @@ -1,7 +1,8 @@ # Native video delivery -The `video-generation` scenario schedules one server job per deployment. H100, -B200 and MI355X SGLang select four-GPU C1 deployments using +The `video-generation` scenario schedules one server job per deployment. SGLang +selects C1 deployments on 1, 2, 4 and 8 B200 GPUs (Ulysses across all GPUs) and +on 4 H100 or MI355X GPUs, all using `h3-v2.yaml`: 20 distinct measured prompts, a separate warmup, 49 denoising updates, and 124 frames at 24 fps and 1344×768. A five-second request produces 124/24 seconds of video. Config presence is not GPU or quality qualification. diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 1db54f77c1..9e141416db 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12262,3 +12262,10 @@ - "Kimi-K3.1-Pro B300 vLLM: enable the recipe's low-latency decode (decode_backend=latency_specialized) at c1-c4, whose decode graphs stay within 64 rows (the image's wider MoE needs 152 resident CTAs, more than a B300's 148 SMs), and run c1-c2 at plain TP8 instead of TP8 x DCP8." - "Match the SGLang recipe's draft depths (DSpark 7 tokens through c8, 6 at c16, 5 at c24-c40, 4 from c48), size each decode graph cap to max-num-seqs x (1 + drafts), and raise the engine-ready timeout and health check to 7200 s." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4833 + +- config-keys: + - h3-bf16-b200-sglang-video + scenario-type: [video-generation] + description: + - "Sweep H3 C1 latency on 1, 2, 4 and 8 B200 GPUs with SGLang Ulysses across all GPUs; the B200 recipe now sets the Ulysses degree from the GPU count, which keeps the 4-GPU point unchanged." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4852 From dd085d0444c467a811df9f7a5c1e82f6c2b5eeac Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Fri, 6 Nov 2026 01:54:48 -0800 Subject: [PATCH 006/111] =?UTF-8?q?[VideoGenX]=20sweep=20H3=20latency=20on?= =?UTF-8?q?=202,=204=20and=208=20H100=20GPUs=20/=20[VideoGenX]=20=E5=9C=A8?= =?UTF-8?q?=202=E3=80=814=E3=80=818=20=E5=BC=A0=20H100=20=E4=B8=8A?= =?UTF-8?q?=E6=89=AB=E6=8F=8F=20H3=20=E5=BB=B6=E8=BF=9F=20(#4857)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: sweep H3 video on 2, 4 and 8 H100 GPUs * chore: link H100 GPU-count changelog entry to #4857 --- .../serving-client-tp2-ulysses4.yaml | 65 +++++++++++++++++++ .../sglang/h100-bf16/serving-client-tp2.yaml | 65 +++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 6 ++ inferencex-e2e/docs/MODELS.md | 2 +- inferencex-e2e/docs/video-generation.md | 5 +- inferencex-e2e/perf-changelog.yaml | 7 ++ 6 files changed, 147 insertions(+), 3 deletions(-) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2-ulysses4.yaml create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2-ulysses4.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2-ulysses4.yaml new file mode 100644 index 0000000000..c09ae2077a --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2-ulysses4.yaml @@ -0,0 +1,65 @@ +schema: 2 +name: h3-sglang-serving-client + +resources: + gpu_type: h100 + +services: + - name: video-server + type: generic + nodes: 1 + critical: true + start: before_workers + command: [python3, -m, infx.bench, video, serve] + args: + - --model-view + - snapshot + - -- + - /opt/sglang/bin/python3 + - -m + - evaluator.srt_selected + - serve + - --model-type + - diffusion + - --model-path + - "{model_path}" + - --model-id + - "{model_id}" + - --revision + - "{model_revision}" + - --model-variant + - fl2va + - --num-gpus + - "{gpus_per_node}" + - --ulysses-degree + - "4" + - --tp-size + - "2" + - --encoder-parallel + - auto + - --warmup-resolutions + - 1344x768 + - --disable-conditioning-cache + - --performance-mode + - speed + - --host + - 0.0.0.0 + - --port + - "30283" + - --enable-torch-compile + - "false" + - --dit-cpu-offload + - "false" + env: + PYTHONPATH: /video-package:/infmax-workspace + PYTHONUNBUFFERED: "1" + VIDEO_SERVER_PORT: "30283" + VIDEO_RUNTIME_REVISION: 297e54fd105e974cdf401e5b4d20253a22c16f90 + HF_HUB_OFFLINE: "1" + TRANSFORMERS_OFFLINE: "1" + readiness: + http: + port: 30283 + path: /health + timeout_seconds: 1800 + interval_seconds: 5 diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2.yaml new file mode 100644 index 0000000000..c37238dee1 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h100-bf16/serving-client-tp2.yaml @@ -0,0 +1,65 @@ +schema: 2 +name: h3-sglang-serving-client + +resources: + gpu_type: h100 + +services: + - name: video-server + type: generic + nodes: 1 + critical: true + start: before_workers + command: [python3, -m, infx.bench, video, serve] + args: + - --model-view + - snapshot + - -- + - /opt/sglang/bin/python3 + - -m + - evaluator.srt_selected + - serve + - --model-type + - diffusion + - --model-path + - "{model_path}" + - --model-id + - "{model_id}" + - --revision + - "{model_revision}" + - --model-variant + - fl2va + - --num-gpus + - "{gpus_per_node}" + - --ulysses-degree + - "1" + - --tp-size + - "2" + - --encoder-parallel + - auto + - --warmup-resolutions + - 1344x768 + - --disable-conditioning-cache + - --performance-mode + - speed + - --host + - 0.0.0.0 + - --port + - "30283" + - --enable-torch-compile + - "false" + - --dit-cpu-offload + - "false" + env: + PYTHONPATH: /video-package:/infmax-workspace + PYTHONUNBUFFERED: "1" + VIDEO_SERVER_PORT: "30283" + VIDEO_RUNTIME_REVISION: 297e54fd105e974cdf401e5b4d20253a22c16f90 + HF_HUB_OFFLINE: "1" + TRANSFORMERS_OFFLINE: "1" + readiness: + http: + port: 30283 + path: /health + timeout_seconds: 1800 + interval_seconds: 5 diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 40db965f44..69eaaa2b15 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7479,9 +7479,15 @@ h3-bf16-h100-sglang-video: - spec: h3-v2.yaml set: release-v1 search-space: + - gpus: 2 + srt-recipe: serving-client-tp2.yaml + conc-list: [1] - gpus: 4 srt-recipe: serving-client.yaml conc-list: [1] + - gpus: 8 + srt-recipe: serving-client-tp2-ulysses4.yaml + conc-list: [1] h3-bf16-b200-sglang-video: image: lmsysorg/sglang:v0.5.23@sha256:f7876bef696ccf9cf75a2bf7ccb82970f5fac577f082fb8ca95c9a59d2160eb5 diff --git a/inferencex-e2e/docs/MODELS.md b/inferencex-e2e/docs/MODELS.md index 404327641b..2255d5898c 100644 --- a/inferencex-e2e/docs/MODELS.md +++ b/inferencex-e2e/docs/MODELS.md @@ -36,7 +36,7 @@ For non-grandfathered models, `ROCM_QUICK_REDUCE_QUANTIZATION`, `VLLM_ROCM_QUICK |---|---|---| | Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see the srt-slurm recipes under [`benchmarks/single_node/srt-slurm-recipes/`](../benchmarks/single_node/srt-slurm-recipes) and the shared client [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation policy](#deprecation-policy)). | | Single-turn 8k1k | 8192 / 1024 | Active. This is the primary fixed-sequence-length scenario. | -| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 GPUs and 4 H100 or MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | +| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 GPUs, 2, 4 and 8 H100 GPUs and 4 MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | | Single-turn 1k1k | 1024 / 1024 | Deprecated since 2026-07-17 ([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Archived configs lived in `configs/deprecated/` until it was deleted (#3463); see git history. The GLM-5.1 B200 TileRT point added later in [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) was the last 1k1k config; it was retired on 2026-10-01 ([#3638](https://github.com/SemiAnalysisAI/InferenceX/pull/3638)). | | Single-turn 1k8k | 1024 / 8192 | **Deprecated for all models** since 2026-03-27 ([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Configs were removed, not archived. | diff --git a/inferencex-e2e/docs/video-generation.md b/inferencex-e2e/docs/video-generation.md index 090962dcb5..4b51acd3b8 100644 --- a/inferencex-e2e/docs/video-generation.md +++ b/inferencex-e2e/docs/video-generation.md @@ -1,8 +1,9 @@ # Native video delivery The `video-generation` scenario schedules one server job per deployment. SGLang -selects C1 deployments on 1, 2, 4 and 8 B200 GPUs (Ulysses across all GPUs) and -on 4 H100 or MI355X GPUs, all using +selects C1 deployments on 1, 2, 4 and 8 B200 GPUs (Ulysses across all GPUs), on +2, 4 and 8 H100 GPUs (TP2, since the model does not fit one 80 GB GPU) and on 4 +MI355X GPUs, all using `h3-v2.yaml`: 20 distinct measured prompts, a separate warmup, 49 denoising updates, and 124 frames at 24 fps and 1344×768. A five-second request produces 124/24 seconds of video. Config presence is not GPU or quality qualification. diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 9e141416db..fbbe353571 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12269,3 +12269,10 @@ description: - "Sweep H3 C1 latency on 1, 2, 4 and 8 B200 GPUs with SGLang Ulysses across all GPUs; the B200 recipe now sets the Ulysses degree from the GPU count, which keeps the 4-GPU point unchanged." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4852 + +- config-keys: + - h3-bf16-h100-sglang-video + scenario-type: [video-generation] + description: + - "Sweep H3 C1 latency on 2, 4 and 8 H100 GPUs with SGLang TP2 and Ulysses 1, 2 and 4; H100 starts at 2 GPUs because the model does not fit one 80 GB GPU, and the 4-GPU point keeps its recipe." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4857 From 41cc713af82bc69204f572e03c2dd8f60485e6ac Mon Sep 17 00:00:00 2001 From: Cameron Quilici Date: Fri, 6 Nov 2026 04:30:22 -0600 Subject: [PATCH 007/111] [K3.1-Pro] Day-0 support vLLM on GB200 (#4799) * feat(kimik3.1): add day-0 vLLM AgentX on GB200 Aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 Kimi K3.1 Pro points on gb200-nv with the K3.1 DSpark draft and Simple CPU KV offload, plus the lane's shared run root and AgentX power rule. * chore(kimik3.1): link the GB200 vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Remove settings that no active recipe uses and vLLM's K3.1 recipe does not document: the DCP comm backend, prompt-token details, and the FlashInfer all-reduce, execute-timeout, UCX and AIPerf failure-threshold env vars. * fix(kimik3.1): pin Dynamo #18124 and use the MXFP4 indexer on GB200 vLLM PD Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL. * fix(kimik3.1): front the GB200 vLLM PD row with the vLLM Router No published Dynamo frontend loads the kimi_k31 tokenizer and the K3.1 Dynamo branch does not build, so the 1P1D row runs direct vLLM workers behind vllm-router 0.1.14 with the same engine arguments, NIXL transfer and MXFP4 indexer. The GB200 Kimi-K3.1 AgentX power rule now also covers vllm. * fix(kimik3.1): label the GB200 vLLM rows as vllm Neither GB200 row runs Dynamo: the aggregated row uses the direct vLLM frontend and the PD row the vLLM Router. Name both keys and their framework vllm, drop the Dynamo-only recipe settings, and scope the Kimi-K3.1 AgentX power rule to vllm. * fix(kimik3.1): keep GB200 vLLM PD worker connections alive for the router vllm-router reuses pooled connections that Uvicorn closes after its 5 s default keep-alive, and with retries disabled the reset aborts the request. Hold PD worker connections for 600 s, as main's GB200 vLLM PD recipe does. * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-gb200-vllm): run disagg decode with expert parallelism Decode at TP16/EP1 holds 131.5 GiB/GPU of weights and leaves 12-19 GiB of KV, less than the 18.17 GiB one 1M-token request needs at c32 and c48. EP16, as on prefill, cuts decode weights to about 103.5 GiB/GPU. --- .../gb200-fp4/agentx/agg-tp8pp2-variants.yaml | 135 +++++++++++++++ .../gb200-fp4/agentx/disagg-variants.yaml | 156 ++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 113 +++++++++++++ inferencex-e2e/perf-changelog.yaml | 9 + 4 files changed, 413 insertions(+) create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/disagg-variants.yaml diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml new file mode 100644 index 0000000000..ef97d6303d --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/agg-tp8pp2-variants.yaml @@ -0,0 +1,135 @@ +schema: 2 + +base: + identity: + model: {} + container: {} + + slurm: + time_limit: "8:00:00" + + health_check: + interval_seconds: 10 + max_attempts: 720 + + resources: + gpu_type: "gb200" + gpus_per_node: 4 + + environment: + # Identical prefixes must hash to identical block keys run-to-run. + PYTHONHASHSEED: "0" + + frontend: + type: vllm + enable_multiple_frontends: false + + engine: + type: vllm + connector: + + roles: + agg: + nodes: 4 + workers: 1 + gpus: 16 + env: + VLLM_ENGINE_READY_TIMEOUT_S: "3600" + VLLM_RPC_TIMEOUT: "600000" + NCCL_CUMEM_ENABLE: "1" + NCCL_MNNVL_ENABLE: "1" + NCCL_NVLS_ENABLE: "1" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + args: + served-model-name: "moonshotai/Kimi-K3.1-Pro" + # One TP8 group (~1.5 TB) is below the checkpoint's 1874 GB minimum on GB200, so PP2 + # splits the layers across two TP8 groups; DCP must equal TP for K3.1. + tensor-parallel-size: 8 + pipeline-parallel-size: 2 + decode-context-parallel-size: 8 + trust-remote-code: true + language-model-only: true + load-format: fastsafetensors + moe-backend: auto + gpu-memory-utilization: 0.88 + max-model-len: 1048576 + max-num-batched-tokens: 8192 + kv-cache-dtype: fp8 + mamba-ssm-cache-dtype: float32 + enable-prefix-caching: true + enable-auto-tool-choice: true + tool-call-parser: kimi_k3 + reasoning-parser: kimi_k3 + # 112742379520 bytes per rank is the dram-utilization 0.5 share of a 4-GPU tray. + kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":112742379520,"lazy_offload":false}}' + + srun_options: + container-remap-root: "" + + telemetry: + collector_join_timeout_seconds: 12 + + benchmark: + placement: + node: head + env: + AIPERF_SERVER_METRICS_URLS: "http://localhost:8000/metrics" + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" + +# Admission is 2x CONC. Graph capture covers max-num-seqs x (1 + draft tokens). +override_tp8pp2_c1: + roles: + agg: + args: + max-num-seqs: 2 + max-cudagraph-capture-size: 12 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c4: + roles: + agg: + args: + max-num-seqs: 8 + max-cudagraph-capture-size: 48 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c8: + roles: + agg: + args: + max-num-seqs: 16 + max-cudagraph-capture-size: 96 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c24: + roles: + agg: + args: + max-num-seqs: 48 + max-cudagraph-capture-size: 240 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c32: + roles: + agg: + args: + max-num-seqs: 64 + max-cudagraph-capture-size: 320 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c48: + roles: + agg: + args: + max-num-seqs: 96 + max-cudagraph-capture-size: 480 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + +override_tp8pp2_c56: + roles: + agg: + args: + max-num-seqs: 112 + max-cudagraph-capture-size: 560 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/disagg-variants.yaml new file mode 100644 index 0000000000..3abb0e28e5 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb200-fp4/agentx/disagg-variants.yaml @@ -0,0 +1,156 @@ +schema: 2 + +base: + identity: + model: {} + container: {} + + slurm: + time_limit: "6:00:00" + + health_check: + max_attempts: 720 + interval_seconds: 10 + + resources: + gpu_type: "gb200" + gpus_per_node: 4 + + environment: + # Identical prefixes must hash to identical block keys run-to-run. + PYTHONHASHSEED: "0" + + frontend: + type: vllm-router + enable_multiple_frontends: false + args: + policy: consistent_hash + prometheus-host: 127.0.0.1 + prometheus-port: 18000 + request-timeout-secs: 14400 + disable-retries: true + + setup_script: vllm-router.sh + + engine: + type: vllm + connector: + + roles: + prefill: + nodes: 4 + workers: 1 + gpus: 16 + env: &kimi_env + VLLM_ENGINE_READY_TIMEOUT_S: "3600" + VLLM_RPC_TIMEOUT: "600000" + # Outlast the router's pooled-connection idle gaps (disable-retries makes a reset fatal). + VLLM_HTTP_TIMEOUT_KEEP_ALIVE: "600" + # NIXL moves the KDA short-conv state in three reads, which needs the DS layout. + VLLM_SSM_CONV_STATE_LAYOUT: "DS" + NCCL_CUMEM_ENABLE: "1" + NCCL_MNNVL_ENABLE: "1" + NCCL_NVLS_ENABLE: "1" + UCX_MEMTYPE_CACHE: "n" + UCX_MEMTYPE_REG_WHOLE: "n" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + args: + # 112742379520 bytes per rank is the dram-utilization 0.5 share of a 4-GPU tray. + kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda"},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":112742379520,"lazy_offload":false}}]}}' + served-model-name: "moonshotai/Kimi-K3.1-Pro" + # NixlConnector rejects pipeline parallelism on KDA hybrids, so each role is one TP16 group. + tensor-parallel-size: 16 + enable-expert-parallel: true + trust-remote-code: true + language-model-only: true + load-format: fastsafetensors + moe-backend: auto + gpu-memory-utilization: 0.88 + max-model-len: 1048576 + max-num-batched-tokens: 16384 + kv-cache-dtype: fp8 + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: float32 + enable-prefix-caching: true + enable-auto-tool-choice: true + tool-call-parser: kimi_k3 + reasoning-parser: kimi_k3 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + decode: + nodes: 4 + workers: 1 + gpus: 16 + env: + <<: *kimi_env + args: + kv-transfer-config: '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda"}' + served-model-name: "moonshotai/Kimi-K3.1-Pro" + tensor-parallel-size: 16 + # TP16/EP1 holds 131.5 GiB/GPU of weights and leaves under 20 GiB of KV for a 1M-token context. + enable-expert-parallel: true + trust-remote-code: true + language-model-only: true + load-format: fastsafetensors + moe-backend: auto + gpu-memory-utilization: 0.9 + max-model-len: 1048576 + # Bounds the MoE autotune workspace to decode-sized batches. + max-num-batched-tokens: 1024 + kv-cache-dtype: fp8 + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: float32 + enable-prefix-caching: true + enable-auto-tool-choice: true + tool-call-parser: kimi_k3 + reasoning-parser: kimi_k3 + speculative-config: '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + + sbatch_directives: + mem: "0" + cpus-per-task: "144" + + srun_options: + mem: "0" + container-remap-root: "" + + telemetry: + collector_join_timeout_seconds: 12 + + benchmark: + placement: + node: head + env: + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" + AIPERF_HTTP_TCP_USER_TIMEOUT: "900000" + +# Admission is 2x CONC on both roles; decode graphs capture max-num-seqs x (1 + 4 draft tokens). +override_1p1d_tp16_c16: + roles: + prefill: + args: + max-num-seqs: 32 + decode: + args: + max-num-seqs: 32 + max-cudagraph-capture-size: 160 + +override_1p1d_tp16_c32: + roles: + prefill: + args: + max-num-seqs: 64 + decode: + args: + max-num-seqs: 64 + max-cudagraph-capture-size: 320 + +override_1p1d_tp16_c48: + roles: + prefill: + args: + max-num-seqs: 96 + decode: + args: + max-num-seqs: 96 + max-cudagraph-capture-size: 480 diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 69eaaa2b15..2b8c2ade35 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7697,6 +7697,119 @@ minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg: prefill: { num-worker: 2, tp: 4, ep: 4, dp-attn: true } decode: { num-worker: 1, tp: 8, ep: 8, dp-attn: true } +kimik3.1-fp4-gb200-vllm-agentic-dspark-tp8pp2-agg: + image: vllm/vllm-openai:kimi-k31-arm64-cu130@sha256:52b793567651780dc529647a211a3d15cec11ebc872ac7492636d7d75340eab4 + model: moonshotai/Kimi-K3.1-Pro + model-prefix: kimik3.1 + runner: cluster:gb200-nv + precision: fp4 + framework: vllm + multinode: true + disagg: false + srt-recipe-dir: kimik3.1/vllm/gb200-fp4/agentx + scenarios: + agentic-coding: + - dram-utilization: 0.5 + search-space: + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [1] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c1 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [4] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c4 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [8] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c8 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [24] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c24 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [32] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c32 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [48] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c48 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [56] + num-nodes: 4 + srt-recipe: agg-tp8pp2-variants.yaml:override_tp8pp2_c56 + power: true + worker: { num-worker: 1, tp: 8, pp: 2, dcp-size: 8, ep: 1, dp-attn: false } + +kimik3.1-fp4-gb200-vllm-agentic-dspark-tp16-disagg: + image: vllm/vllm-openai:kimi-k31-arm64-cu130@sha256:52b793567651780dc529647a211a3d15cec11ebc872ac7492636d7d75340eab4 + model: moonshotai/Kimi-K3.1-Pro + model-prefix: kimik3.1 + runner: cluster:gb200-nv + precision: fp4 + framework: vllm + router: { name: vllm-router, version: "0.1.14" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + srt-recipe-dir: kimik3.1/vllm/gb200-fp4/agentx + scenarios: + agentic-coding: + # Host offload runs on the prefill role only; decode keeps plain NIXL. + - dram-utilization: 0.5 + search-space: + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [16] + srt-recipe: disagg-variants.yaml:override_1p1d_tp16_c16 + power: true + prefill: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [32] + srt-recipe: disagg-variants.yaml:override_1p1d_tp16_c32 + power: true + prefill: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: vllm-simple } + conc-list: [48] + srt-recipe: disagg-variants.yaml:override_1p1d_tp16_c48 + power: true + prefill: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false } + minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg: image: vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d76347628973fb80519a919a2f2740b6645b62cba3fe2bd87b25f111bd0e948 model: MiniMaxAI/MiniMax-M3.1-Flash diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index fbbe353571..c730b50783 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12276,3 +12276,12 @@ description: - "Sweep H3 C1 latency on 2, 4 and 8 H100 GPUs with SGLang TP2 and Ulysses 1, 2 and 4; H100 starts at 2 GPUs because the model does not fit one 80 GB GPU, and the 4-GPU point keeps its recipe." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4857 + +- config-keys: + - kimik3.1-fp4-gb200-vllm-agentic-dspark-tp8pp2-agg + - kimik3.1-fp4-gb200-vllm-agentic-dspark-tp16-disagg + scenario-type: + - agentic-coding + description: + - "Add day-0 Kimi K3.1 Pro vLLM AgentX on GB200: aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 + EP16 with DSpark (K3.1 golden AL curve) and Simple CPU KV offload." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4799 From 1e7c8c9781bde493c4262097c67717cb6f5affd0 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Fri, 6 Nov 2026 06:47:41 -0500 Subject: [PATCH 008/111] =?UTF-8?q?Add=20GLM-5.3-Flash=20FP8=20MI355X=20SG?= =?UTF-8?q?Lang=20AgentX=20on=20the=20cookbook=20recipe=20/=20=E6=96=B0?= =?UTF-8?q?=E5=A2=9E=E5=9F=BA=E4=BA=8E=20cookbook=20=E9=85=8D=E6=96=B9?= =?UTF-8?q?=E7=9A=84=20GLM-5.3-Flash=20FP8=20MI355X=20SGLang=20AgentX=20(#?= =?UTF-8?q?4720)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 MI355X lane * chore(bench): record native Flash MI355X sweep provenance * fix(ci): preserve canary gates after skipped reuse checks * fix(bench): disable integer QuickReduce for Flash MI355X * fix(bench): verify pinned Flash cache before offline serving * fix(bench): pin Flash serving tokenizer to staged snapshot * fix(bench): restore canonical Flash client tokenizer lookup * fix(bench): clear Flash client offline flags for tokenizer warmup * fix(bench): use upstream Flash hybrid attention fix on MI355X * feat(bench): consolidate GLM-5.3-Flash FP8 MI355X onto the SGLang cookbook recipe Rebuild the MI355X FP8 lane on current main with the cookbook MI355X FP8 Low Latency recipe: TP4/EP4, HBM-only KV at conc 1-32 and the cookbook HiCache L1 + L2 tier at conc 40, 48 and 64. * fix(bench): record measured GLM-5.3-Flash MI355X HiCache DRAM and page size The c40/c48 HiCache servers allocate 187.33 GB of host memory per rank (749.33 GB per TP4 server); record 751 GB. SGLang v0.5.22 forces page size 256 for DSA models, so state it. * docs(bench): state the measured MI355X HiCache host pools per rank * chore: refresh PR #4720 for sweep reuse [skip-sweep] --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> --- .../sglang/mi355x-fp8-mtp/agentic.yaml | 134 ++++++++++++++++++ inferencex-e2e/configs/amd-master.yaml | 21 +++ .../infx/launch/drivers/srt/models.py | 1 + inferencex-e2e/perf-changelog.yaml | 8 ++ 4 files changed, 164 insertions(+) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/mi355x-fp8-mtp/agentic.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/mi355x-fp8-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/mi355x-fp8-mtp/agentic.yaml new file mode 100644 index 0000000000..2436f7b4a7 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/mi355x-fp8-mtp/agentic.yaml @@ -0,0 +1,134 @@ +# SGLang cookbook GLM-5.3-Flash MI355X FP8 Low Latency recipe (TP4/EP4, MTP 5/1/6) +# with the default deploy-panel overlays; the dram points add its HiCache L1 + L2 tier. +base: + schema: 2 + name: glm5.3flash-fp8-mi355x-sglang-agentic + resources: + gpu_type: mi355x + gpus_per_node: 8 + frontend: + type: sglang + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: sglang + roles: + agg: + nodes: 1 + workers: 1 + gpus: 4 + args: + served-model-name: zai-org/GLM-5.3-Flash + trust-remote-code: true + quantization: fp8 + tensor-parallel-size: 4 + expert-parallel-size: 4 + # The cookbook passes 64; v0.5.22 forces 256 for DSA models. + page-size: 256 + mem-fraction-static: 0.82 + max-running-requests: 96 + cuda-graph-max-bs-decode: 96 + min-free-slots-delay: 1 + chunked-prefill-size: 8192 + max-prefill-tokens: 8192 + prefill-decode-interval: 0 + dsa-prefill-backend: tilelang + dsa-decode-backend: tilelang + linear-attn-verify-backend: triton + kv-cache-dtype: fp8_e4m3 + moe-runner-backend: aiter + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + speculative-attention-mode: prefill + speculative-accept-threshold-single: 1.0 + speculative-accept-threshold-acc: 1.0 + speculative-moe-a2a-backend: none + mm-feature-transport: cpu + reasoning-parser: auto + tool-call-parser: auto + watchdog-timeout: 1800 + dist-timeout: 600 + enable-metrics: true + env: + PYTHONNOUSERSITE: '1' + # Outlast AIPerf's pooled connections past Uvicorn's keep-alive. + SGLANG_TIMEOUT_KEEP_ALIVE: '900' + SGLANG_USE_AITER: '1' + AITER_ONLINE_TUNE: '0' + SGLANG_OPT_FUSED_KDA_VERIFY: '1' + SGLANG_DSA_FUSE_TOPK: '0' + SGLANG_ENABLE_WAR_BARRIER: '1' + SGLANG_FORCE_COARSE_WAR_BARRIER: '1' + # The image sets INT8; MODELS.md's all-reduce policy requires NONE for new models. + ROCM_QUICK_REDUCE_QUANTIZATION: NONE + benchmark: + env: + # Warmup can leave bytes unacknowledged past AIPerf's 30 s default. + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' + +override_tp4_c1: + benchmark: + env: + CONC: '1' + KV_OFFLOADING: none + +override_tp4_c2: + benchmark: + env: + CONC: '2' + KV_OFFLOADING: none + +override_tp4_c8: + benchmark: + env: + CONC: '8' + KV_OFFLOADING: none + +override_tp4_c16: + benchmark: + env: + CONC: '16' + KV_OFFLOADING: none + +override_tp4_c24: + benchmark: + env: + CONC: '24' + KV_OFFLOADING: none + +override_tp4_c32: + benchmark: + env: + CONC: '32' + KV_OFFLOADING: none + +# hicache-size 180 per rank; run 42074044563 allocated 187.33 GB per rank (113.10 GB KV, +# 66.94 GB KDA state, 7.29 GB DSA indexer), 749.33 GB for the TP4 server. +override_tp4_c40_hicache: + roles: + agg: + args: + enable-hierarchical-cache: true + hicache-size: 180 + hicache-write-policy: write_through + benchmark: + env: + CONC: '40' + KV_OFFLOADING: dram + +override_tp4_c48_hicache: + roles: + agg: + args: + enable-hierarchical-cache: true + hicache-size: 180 + hicache-write-policy: write_through + benchmark: + env: + CONC: '48' + KV_OFFLOADING: dram diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 04775a3909..9e9e23dc1a 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -1002,6 +1002,27 @@ glm5.3-fp8-mi355x-tilert-agentic: ep: 1 dp-attn: false +# SGLang cookbook MI355X FP8 Low Latency recipe on four GPUs: HBM-only KV up to +# c32, the cookbook HiCache L1 + L2 tier above it. +glm5.3flash-fp8-mi355x-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.22-rocm10-mi35x@sha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f + model: zai-org/GLM-5.3-Flash + model-prefix: glm5.3flash + runner: cluster:mi355x-amds + precision: fp8 + framework: sglang + multinode: false + srt-recipe-dir: glm5.3flash/sglang/mi355x-fp8-mtp + scenarios: + agentic-coding: + # HiCache allocates 187.33 GB of host memory per rank (113.10 GB KV, 66.94 GB KDA + # state, 7.29 GB DSA indexer), 749.33 GB for the TP4 server; dram-utilization 0.501 + # -> TOTAL_CPU_DRAM_GB = 2861022 MiB cap * 0.501 * 4/8 = 716,686 MiB = 751 GB. + - dram-utilization: 0.501 + search-space: + - { tp: 4, ep: 4, kv-offloading: none, conc-list: [1, 2, 8, 16, 24, 32], spec-decoding: mtp, srt-recipe: agentic.yaml } + - { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [40, 48], spec-decoding: mtp, srt-recipe: agentic.yaml } + # SGLang ROCm 10 MI35X release image. TP2/EP1 up to c32, TP4/EP1 up to c16, # DP2 attention with TP MoE at c32 and c64 (c32 overlaps TP2). Host Engram on all. dsv41flash-fp4-mi355x-sglang-agentic-dspark: diff --git a/inferencex-e2e/infx/launch/drivers/srt/models.py b/inferencex-e2e/infx/launch/drivers/srt/models.py index b3e5b47a1e..68747974c5 100644 --- a/inferencex-e2e/infx/launch/drivers/srt/models.py +++ b/inferencex-e2e/infx/launch/drivers/srt/models.py @@ -194,6 +194,7 @@ def single_node_model_path(cluster: Cluster, request: LaunchRequest) -> str: Match(agentic=True, model_glob="MiniMaxAI/MiniMax-M3.1-*"), Match(agentic=True, model_glob="amd/MiniMax-M3.1-*"), Match(agentic=True, model_glob="zai-org/GLM-5.2-FP8"), + Match(agentic=True, model_glob="zai-org/GLM-5.3-Flash"), Match(agentic=True, model_glob="deepseek-ai/DeepSeek-V4.1-Flash"), Match(agentic=True, model_glob="amd/Muse-Spark-Open-*"), Match(model_glob="MiniMaxAI/MiniMax-H3"), diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index c730b50783..42401e2a75 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12285,3 +12285,11 @@ description: - "Add day-0 Kimi K3.1 Pro vLLM AgentX on GB200: aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 + EP16 with DSpark (K3.1 golden AL curve) and Simple CPU KV offload." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4799 + +- config-keys: + - glm5.3flash-fp8-mi355x-sglang-agentic-mtp + scenario-type: [agentic-coding] + description: + - "Add native FP8 GLM-5.3-Flash AgentX on MI355X with the official SGLang v0.5.22 ROCm image pinned by digest and the SGLang cookbook MI355X FP8 Low Latency recipe: TP4/EP4, AITER MoE, FP8 KV with TileLang DSA, embedded NextN MTP 5/1/6 served as shipped." + - "HBM-only KV at conc 1, 2, 8, 16, 24 and 32; the cookbook HiCache L1 + L2 tier (180 GB per rank, write-through) at conc 40 and 48 recording its 749.33 GB host allocation (187.33 GB per rank) as a 751 GB CPU DRAM share. ROCM_QUICK_REDUCE_QUANTIZATION=NONE per the all-reduce policy." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4720 From ddf781dde3ec478b01dd7affa97c93526dbd5534 Mon Sep 17 00:00:00 2001 From: Cameron Quilici Date: Fri, 6 Nov 2026 08:57:35 -0600 Subject: [PATCH 009/111] =?UTF-8?q?[K3.1-Pro]=20Day-0=20support=20vLLM=20o?= =?UTF-8?q?n=20MI355X=20/=20[K3.1-Pro]=20MI355X=20=E4=B8=8A=20vLLM=20?= =?UTF-8?q?=E7=9A=84=20Day-0=20=E6=94=AF=E6=8C=81=20(#4806)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(kimik3.1): add Kimi K3.1 Pro vLLM AgentX on MI355X Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification, following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs. GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60. * chore(kimik3.1): link the MI355X vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Write the per-concurrency points as plain override variants instead of zip overrides. * fix(kimik3.1): leave GPU headroom for pinned scratch on MI355X The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38 needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X recipe did on the same nodes. * perf(kimik3.1): match the apex-1104 sweep's points and batching The MI355X AgentX lane and the reference apex-1104 sweep differed in the points they ran and in batching, so a comparison between them could not attribute anything to the engine build. Align the parts that shape the workload: seven points instead of ten (drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs below concurrency as that sweep sets it, max-num-batched-tokens 16384 except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of host DRAM for the Simple CPU offload. Keep this lane's own precision and speculative settings: 5 DSpark tokens with block rejection at every point, the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state, BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and no FP8 indexer override or KDA input-projection quantization. Those are deliberate here, and copying them would change what the runs measure -- synthetic acceptance in particular reports a modelled throughput and makes the output nondeterministic by construction. The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% -- the recipe's cpu_bytes_to_use_per_rank wins. Say so instead. Co-Authored-By: Claude Opus 5 (1M context) * fix(kimik3.1): resolve the MI355X DRAM budget to the recipe's 2577 GB The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to 2577 GB, within the 3 TB per-server limit. Align comments with the settings they describe and correct the changelog: throughput points still run synthetic acceptance at the golden 3.08. * perf(kimik3.1): use 7 DSpark tokens and the specialized decode at conc 1 The conc-1 latency point runs the reference sweep's latency_specialized decode backend with 7 draft tokens; the throughput points stay at 5. K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only and breakable graphs are the only thing that would add the prefill sizes. Turn them off and leave scratch reclaim on, as the image sets it. Co-Authored-By: Claude Opus 5 (1M context) * perf(kimik3.1): prefill with the AITER FA MLA backend at conc 48 only The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves conc 1-32 on the default. Move it out of base into that point's override. Co-Authored-By: Claude Opus 5 (1M context) * perf(kimik3.1): set breakable cudagraphs and scratch reclaim per point The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the conc-1 values into that point's override and leave the base on the sweep's throughput setting. Co-Authored-By: Claude Opus 5 (1M context) * fix(kimik3.1): keep scratch reclaim off on the MI355X DRAM points On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at conc 4-48 and no async scratch limit there; conc 1 keeps its own settings. * perf(kimik3.1): trim the conc-32 and conc-48 points on MI355X Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no successful request. Cut the per-step working set at the two largest points: max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens, with the capture lists rebuilt from those. The other five points keep the reference apex-1104 sweep's settings. Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill backend at its default everywhere. Co-Authored-By: Claude Opus 5 (1M context) * perf(kimik3.1): keep scratch allocated and step the draft count down MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release blocks until the running dispatches finish, which is how the conc-16 and conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a reclaim that no longer happens. Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB that dram-utilization 0.6 resolves to. Describe the points on their own terms rather than against another sweep. Co-Authored-By: Claude Opus 5 (1M context) * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-mi355x-vllm): lower gpu-memory-utilization to 0.86 at conc 16-32 With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest. * fix * fix * perf(kimik3.1): refresh the day-0 image and map the offload pool with huge pages THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the 224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55 million of them to register and keep mapped, against 107 thousand 2 MiB pages. Co-Authored-By: Claude Opus 5 (1M context) * docs(changelog): sync Kimi-K3.1-Pro MI355X vLLM block with the diff [skip-sweep] - Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2). - TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to). - Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48. - Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens. - Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts. - K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE). - Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's. Co-Authored-By: Claude Opus 5.5 * chore: refresh PR #4806 for sweep reuse [skip-sweep] --------- Co-authored-by: billishyahao Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: seungrokj --- .../kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml | 198 ++++++++++++++++++ inferencex-e2e/configs/amd-master.yaml | 18 ++ inferencex-e2e/perf-changelog.yaml | 15 ++ 3 files changed, 231 insertions(+) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml new file mode 100644 index 0000000000..045d0cfb49 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml @@ -0,0 +1,198 @@ +base: + schema: 2 + name: kimik3.1-fp4-mi355x-vllm-agentic + resources: + gpu_type: mi355x + gpus_per_node: 8 + frontend: + type: vllm + enable_multiple_frontends: false + observability: + enabled: false + tachometer: + enabled: false + engine: + type: vllm + connector: null + # A node's first run downloads ~1.56 TB of MXFP4 shards into its local HF cache before loading. + health_check: + interval_seconds: 10 + max_attempts: 720 + roles: + agg: + nodes: 1 + workers: 1 + gpus: 8 + args: + served-model-name: moonshotai/Kimi-K3.1-Pro + tensor-parallel-size: 8 + decode-context-parallel-size: 8 + dcp-comm-backend: a2a + cp-kv-cache-interleave-size: 1 + trust-remote-code: true + moe-backend: auto + seed: 7 + language-model-only: true + load-format: fastsafetensors + enable-auto-tool-choice: true + tool-call-parser: kimi_k3 + reasoning-parser: kimi_k3 + max-model-len: 1048576 + enable-prefix-caching: true + prefix-match-unit: 64 + kv-cache-dtype: fp8 + # The K3.1 week-1 precision policy requires an FP32 KDA recurrent state. + mamba-ssm-cache-dtype: float32 + # The policy's FP4 indexer cache; the image's AITER pads the FP4 indexer pages it needs. + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + additional-config: '{"kda_prefill_backend":"fused"}' + # Prefix-cached prefill chunks end on K3.1's 1152-token packed DSA block, so the budget floors to a multiple of 1152. + max-num-batched-tokens: 16384 + # Kernel scratch, up to HSA_SCRATCH_SINGLE_LIMIT_ASYNC per dispatch, is allocated outside this budget, so leave headroom. + gpu-memory-utilization: 0.90 + stream-interval: 10 + disable-uvicorn-access-log: true + # 224.875 GB per rank; TOTAL_CPU_DRAM_GB (1799) split across the 8 TP ranks. + kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":224875000000,"lazy_offload":false}}' + env: + VLLM_ROCM_USE_AITER: '1' + VLLM_ROCM_USE_AITER_MOE_SITUV2: '1' + SAFETENSORS_FAST_GPU: '1' + VLLM_USE_V2_MODEL_RUNNER: '1' + VLLM_FASTSAFETENSORS_QUEUE_SIZE: '-1' + # K3.1 has no torch.compile submodule, so only breakable graphs capture the prefill + # sizes the throughput points list. The conc-1 override turns this off. + VLLM_USE_BREAKABLE_CUDAGRAPH: '1' + VLLM_ROCM_AITER_MLA_ASM_PADDING: asm + # Verify the draft tokens with the AITER assembly DCP kernel. + VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm + VLLM_ROCM_DCP_AITER_ALLGATHER: '1' + VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1' + # AITER reads its own variable for the same kernels. + AITER_FLYDSL_SPARSE_MLA: '1' + VLLM_ROCM_DSA_DCP_GLOBAL_KEY_PREFILL: '1' + VLLM_ROCM_DSA_DCP_ROWSHARD_PREFILL: '1' + VLLM_USE_DIRECT_DCP_A2A: '0' + VLLM_USE_DIRECT_DCP_Q_GATHER: '0' + VLLM_USE_DIRECT_DCP_KV_GATHER: '0' + # The K3.1 policy and MODELS.md keep all-reduce in BF16, so quick reduce must not quantize. + VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE + AITER_FLYDSL_STAGE2_FP8: '0' + AITER_BF16_FP8_MOE_BOUND: '0' + HSA_NO_SCRATCH_RECLAIM: '1' + # Back the offload pool with transparent huge pages. Torch's CPU allocator + # only calls madvise(MADV_HUGEPAGE) when this is set, and the host is in + # madvise mode, so without it the pool is all 4 KiB pages. + THP_MEM_ALLOC_ENABLE: '1' + VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: '1200' + PYTHONNOUSERSITE: '1' + VLLM_ENGINE_READY_TIMEOUT_S: '7200' + # Identical prefixes must hash to identical block keys run-to-run. + PYTHONHASHSEED: '42' + benchmark: + env: + AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:' + +# max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. The capture list is +# the decode sizes, multiples of one plus the draft token count, then the prefill ladder. +# Throughput runs replace block rejection with the golden acceptance length. +override_gpu_kv_c1: + roles: + agg: + args: + max-num-seqs: 1 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[2,4,6,8,10,12,14,16]}' + additional-config: '{"decode_backend":"latency_specialized","kda_prefill_backend":"fused"}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + kv-transfer-config: null + env: + # This point captures only the decode sizes, so breakable graphs buy nothing. + VLLM_USE_BREAKABLE_CUDAGRAPH: '0' + benchmark: + env: + CONC: '1' + KV_OFFLOADING: 'none' + +override_dram_c4: + roles: + agg: + args: + max-num-seqs: 2 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '4' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + +override_dram_c10: + roles: + agg: + args: + max-num-seqs: 6 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '10' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + +override_dram_c16: + roles: + agg: + args: + max-num-seqs: 10 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '16' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + +override_dram_c24: + roles: + agg: + args: + max-num-seqs: 16 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '24' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + +override_dram_c32: + roles: + agg: + args: + max-num-seqs: 24 + max-num-batched-tokens: 10240 + gpu-memory-utilization: 0.91 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '32' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + +override_dram_c48: + roles: + agg: + args: + max-num-seqs: 48 + max-num-batched-tokens: 10240 + gpu-memory-utilization: 0.91 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + benchmark: + env: + CONC: '48' + KV_OFFLOADING: 'dram' + TOTAL_CPU_DRAM_GB: '1799' + diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 9e9e23dc1a..8b7a89722a 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -926,6 +926,24 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark: - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32], srt-recipe: agentic.yaml } - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [64], srt-recipe: agentic.yaml:override_tp4_dspark3_c64_offload } +kimik3.1-fp4-mi355x-vllm-agentic-dspark: + image: vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 + model: moonshotai/Kimi-K3.1-Pro + model-prefix: kimik3.1 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + srt-recipe-dir: kimik3.1/vllm/mi355x-fp4-mtp + scenarios: + agentic-coding: + # 0.6 resolves to 1799 GB of host DRAM, which the recipe's Simple CPU offload splits across the 8 TP ranks. + - dram-utilization: 0.6 + search-space: + # The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 keeps the KV cache on the GPUs. + - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml } + - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 10, 16, 24, 32, 48], srt-recipe: agentic.yaml } + # MiniMax-M3.1-Flash AgentX on MI355X with SGLang: day-zero image from the SGLang # cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1), # manifest digest verified 2026-10-24. Serve flags follow the cookbook's diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 42401e2a75..97190a884e 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12035,6 +12035,21 @@ - "Add Kimi-K3.1-Pro day-0 vLLM AgentX on B300: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview) and Simple CPU KV offload, conc 1-70." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4796 +- config-keys: + - kimik3.1-fp4-mi355x-vllm-agentic-dspark + scenario-type: + - agentic-coding + description: + - "Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2)." + - "TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to)." + - "Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48." + - "Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens." + - "Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts." + - "K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE)." + - "Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real." + - "No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4806 + - config-keys: - kimik3.1-fp4-mi355x-sglang-agentic-dflash scenario-type: From f8007b14398a2041ab28d10f1d9ff48e9a6cdf3d Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 00:03:50 +0800 Subject: [PATCH 010/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20MI325X=20vLLM=20AgentX=20DSpark=20image=20to=20v?= =?UTF-8?q?0.33.0=20/=20[Klaud=20Cold]=20=E5=B0=86=20DeepSeek-V4.1-Flash?= =?UTF-8?q?=20FP4=20MI325X=20vLLM=20AgentX=20DSpark=20=E9=95=9C=E5=83=8F?= =?UTF-8?q?=E6=9B=B4=E6=96=B0=E8=87=B3=20v0.33.0=20(#4794)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(dsv41flash): bump MI325X vLLM AgentX DSpark image to v0.33.0 Co-Authored-By: Claude Opus 5.5 * chore(dsv41flash): add MI325X vLLM v0.33.0 perf-changelog entry Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- inferencex-e2e/configs/amd-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 8b7a89722a..07cf9604ba 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -723,7 +723,7 @@ glm5.2-fp8-mi325x-sglang-agentic-mtp: # golden AL 3.51 for throughput, real block verification for evals, adaptive # verification off on ROCm. dsv41flash-fp4-mi325x-vllm-agentic-dspark: - image: vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905 + image: vllm/vllm-openai-rocm:v0.33.0@sha256:4fcd39b758b660c4e9c345c021e2ea8f0ccdb31e3c32507d676cc3005ae83fba model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi325x-amd diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 97190a884e..0b6ca4da99 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12308,3 +12308,10 @@ - "Add native FP8 GLM-5.3-Flash AgentX on MI355X with the official SGLang v0.5.22 ROCm image pinned by digest and the SGLang cookbook MI355X FP8 Low Latency recipe: TP4/EP4, AITER MoE, FP8 KV with TileLang DSA, embedded NextN MTP 5/1/6 served as shipped." - "HBM-only KV at conc 1, 2, 8, 16, 24 and 32; the cookbook HiCache L1 + L2 tier (180 GB per rank, write-through) at conc 40 and 48 recording its 749.33 GB host allocation (187.33 GB per rank) as a 751 GB CPU DRAM share. ROCM_QUICK_REDUCE_QUANTIZATION=NONE per the all-reduce policy." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4720 + +- config-keys: + - dsv41flash-fp4-mi325x-vllm-agentic-dspark + description: + - "Update the vLLM ROCm image from v0.32.0 to v0.33.0, pinned by digest." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4794 + scenario-type: [agentic-coding] From 38c712f09b10fe4c920dfbc85403cfeb41cb209b Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 00:10:27 +0800 Subject: [PATCH 011/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20H100=20vLLM=20AgentX=20DSpark=20image=20to=20nig?= =?UTF-8?q?htly-00b6c4c=20/=20[Klaud=20Cold]=20=E5=B0=86=20DeepSeek-V4.1-F?= =?UTF-8?q?lash=20FP4=20H100=20vLLM=20AgentX=20DSpark=20=E9=95=9C=E5=83=8F?= =?UTF-8?q?=E6=9B=B4=E6=96=B0=E8=87=B3=20nightly-00b6c4c=20(#4569)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(dsv41flash): update H100 vLLM AgentX image to nightly-00b6c4c Co-Authored-By: Claude Opus 5.5 * chore(perf-changelog): add DSv4.1 Flash H100 vLLM AgentX image refresh Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 2b8c2ade35..501307ac71 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5772,7 +5772,7 @@ glm5.3flash-fp4-gb300-sglang-agentic-mtp: # caps batched tokens at 4096. Concurrency is bounded by the measured KV # ceiling rather than a guess; see the search-space comment. dsv41flash-fp4-h100-vllm-agentic-dspark: - image: vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d@sha256:6b90105076216e861cd30fae8828efef9263a0399137b894289f3b6a685bf4ff + image: vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:h100-dsxe diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 0b6ca4da99..bec917d7c0 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12315,3 +12315,10 @@ - "Update the vLLM ROCm image from v0.32.0 to v0.33.0, pinned by digest." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4794 scenario-type: [agentic-coding] + +- config-keys: + - dsv41flash-fp4-h100-vllm-agentic-dspark + description: + - "Update the vLLM image from nightly-e4a5b5537344 to nightly-00b6c4c565a8, pinned by digest." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4569 + scenario-type: [agentic-coding] From f7e72d49960b74fdac8691f49307751f09e970db Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 00:27:33 +0800 Subject: [PATCH 012/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20GB300=20vLLM=20AgentX=20DSpark=20image=20to=20ni?= =?UTF-8?q?ghtly-00b6c4c5=20/=20[Klaud=20Cold]=20=E5=B0=86=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20GB300=20vLLM=20AgentX=20DSpark=20=E9=95=9C?= =?UTF-8?q?=E5=83=8F=E6=9B=B4=E6=96=B0=E8=87=B3=20nightly-00b6c4c5=20(#455?= =?UTF-8?q?8)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * config: update GB300 DSV4.1 Flash vLLM AgentX DSpark image to nightly-00b6c4c5 Co-Authored-By: Claude Opus 5.5 * perf-changelog: add GB300 DSV4.1 Flash vLLM AgentX DSpark nightly-00b6c4c5 entry Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 501307ac71..fe4f5aad00 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5996,7 +5996,7 @@ dsv41flash-fp4-b200-sglang-agentic-dspark: - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [48, 64, 96, 128, 192, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } dsv41flash-fp4-gb300-vllm-agentic-dspark: - image: vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7 + image: vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:gb300-nv diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index bec917d7c0..86f1786a8b 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12322,3 +12322,10 @@ - "Update the vLLM image from nightly-e4a5b5537344 to nightly-00b6c4c565a8, pinned by digest." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4569 scenario-type: [agentic-coding] + +- config-keys: + - dsv41flash-fp4-gb300-vllm-agentic-dspark + description: + - "Update the vLLM image from nightly-dev-arm64-cu130-ac9126e58aa7 to the digest-pinned nightly-00b6c4c5." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4558 + scenario-type: [agentic-coding] From 0f205d2134bacae1ca755e295220c8a285b739ef Mon Sep 17 00:00:00 2001 From: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Date: Fri, 6 Nov 2026 14:07:21 -0500 Subject: [PATCH 013/111] Add Kimi-K3.1-Pro GB300 Dynamo+vLLM disaggregated AgentX configs (#4839) Co-authored-by: Claude Opus 5.5 --- .../vllm/gb300-fp4/agentx/disagg-dep16.yaml | 233 ++++++++++ .../gb300-fp4/agentx/disagg-variants.yaml | 437 ++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 97 ++++ inferencex-e2e/perf-changelog.yaml | 9 + 4 files changed, 776 insertions(+) create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-dep16.yaml create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-variants.yaml diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-dep16.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-dep16.yaml new file mode 100644 index 0000000000..42dced28de --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-dep16.yaml @@ -0,0 +1,233 @@ +schema: 2 + +base: + name: kimik3.1-fp4-gb300-dynamo-vllm-agentx-disagg-dep16 + identity: + model: {} + container: {} + frameworks: + vllm: "0.1.dev24218+g78e9a0c28" + dynamo: "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6" + + # ai-dynamo/dynamo#18124 head: the first Dynamo that registers the kimi_k31 model type. + dynamo: + install: true + source: + rev: "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6" + + # The DSpark draft is read from the shared Hub cache. + environment: + HF_HUB_CACHE: /hf_hub_cache + + slurm: + time_limit: "04:00:00" + + health_check: + max_attempts: 720 + interval_seconds: 10 + + resources: + gpu_type: "gb300" + gpus_per_node: 4 + + services: + - name: etcd + type: etcd + placement: + node: infra + - name: nats + type: nats + placement: + node: infra + options: + max_payload_mb: 32 + - name: mooncake-master + type: mooncake-master + args: + - "--default_kv_lease_ttl=60000" + - "--eviction_high_watermark_ratio=0.95" + - "--eviction_ratio=0.10" + options: + store_config: + metadata_server: "P2PHANDSHAKE" + global_segment_size: "160GB" + local_buffer_size: "4GB" + protocol: "rdma" + device_name: "@fabric.rdma-devices" + mode: "embedded" + enable_offload: false + + frontend: + type: dynamo + enable_multiple_frontends: false + args: + router-mode: "least-loaded" + router-session-affinity-ttl-secs: 1 + env: + DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600" + DYN_TOKENIZER_CACHE_BYTES: "8589934592" + DYN_TCP_REQUEST_TIMEOUT: "120" + DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION: "1" + + engine: + type: vllm + connector: + dp_launch_mode: per_node + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + + roles: + prefill: + nodes: 4 + workers: 1 + gpus: 16 + env: + <<: &kimi_env + DYN_TCP_REQUEST_TIMEOUT: "120" + VLLM_USE_DIRECT_DCP_A2A: "1" + VLLM_USE_DIRECT_DCP_Q_GATHER: "1" + VLLM_USE_DIRECT_DCP_KV_GATHER: "1" + VLLM_ALLREDUCE_USE_FLASHINFER: "1" + VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT: "0" + DYN_REQUEST_PLANE: "tcp" + ETCD_LEASE_TTL: "600" + VLLM_RPC_TIMEOUT: "600000" + TILELANG_CLEANUP_TEMP_FILES: "1" + VLLM_USE_NCCL_SYMM_MEM: "0" + NCCL_CUMEM_ENABLE: "1" + NCCL_MNNVL_ENABLE: "1" + NCCL_NVLS_ENABLE: "1" + VLLM_SERVER_DEV_MODE: "1" + VLLM_USE_V2_MODEL_RUNNER: "1" + VLLM_USE_RUST_FRONTEND: "1" + VLLM_MOONCAKE_LOAD_RECV_THREADS: "4" + # Lookups lease only the store keys a hit loads. + VLLM_MOONCAKE_STORE_SELECTIVE_LEASE: "1" + MC_SLICE_SIZE: "1048576" + MC_MAX_MR_SIZE: "4294967296" + NCCL_P2P_LEVEL: "NVL" + MC_ENABLE_DEST_DEVICE_AFFINITY: "1" + WITH_NVIDIA_PEERMEM: "0" + NCCL_NET_PLUGIN: "none" + UCX_MEMTYPE_REG_WHOLE: "n" + UCX_TCP_AF_PRIO: "inet" + VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" + # NIXL transfers the KDA conv state in the DS layout. + VLLM_SSM_CONV_STATE_LAYOUT: "DS" + TOKENSPEED_MLA_FP16_PARTIALS: "0" + VLLM_ENGINE_READY_TIMEOUT_S: "3600" + VLLM_ENABLE_GEMM_RS: "1" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "1024" + DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "1024" + args: + kv-transfer-config: &kv_transfer '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}' + served-model-name: "moonshotai/Kimi-K3.1-Pro" + prefix-match-unit: 128 + load-format: "fastsafetensors" + kv-cache-dtype: "fp8" + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: "float32" + mamba-cache-dtype: "float32" + gpu-memory-utilization: 0.95 + tensor-parallel-size: 1 + data-parallel-size: 16 + data-parallel-rpc-port: 13355 + enable-expert-parallel: true + moe-backend: deep_gemm_mega_moe + no-enable-flashinfer-autotune: true + enable-cumem-allocator: true + trust-remote-code: true + max-num-seqs: 32 + max-num-batched-tokens: 2048 + # Caps each prompt's chunk so one long prompt does not fill a DP rank's step. + long-prefill-token-threshold: 512 + long-prefill-token-threshold-adaptive: true + max-cudagraph-capture-size: 2048 + compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048]}' + stream-interval: 10 + language-model-only: true + enable-prefix-caching: true + speculative-config: &spec '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","method":"dspark","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + decode: + nodes: 4 + workers: 1 + gpus: 16 + env: + <<: *kimi_env + VLLM_ENABLE_GEMM_RS: "0" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + args: + kv-transfer-config: *kv_transfer + served-model-name: "moonshotai/Kimi-K3.1-Pro" + prefix-match-unit: 128 + load-format: "fastsafetensors" + kv-cache-dtype: "fp8" + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: "float32" + mamba-cache-dtype: "float32" + gpu-memory-utilization: 0.95 + tensor-parallel-size: 1 + data-parallel-size: 16 + data-parallel-rpc-port: 13345 + enable-expert-parallel: true + moe-backend: deep_gemm_mega_moe + no-enable-flashinfer-autotune: true + enable-cumem-allocator: true + trust-remote-code: true + stream-interval: 10 + language-model-only: true + enable-prefix-caching: true + additional-config: '{"decode_backend":"latency_specialized"}' + use-replayssm: true + max-num-seqs: 32 + max-num-batched-tokens: 160 + max-cudagraph-capture-size: 160 + compilation-config: '{"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160]}' + speculative-config: *spec + + sbatch_directives: + mem: "0" + # 72 pins every vLLM worker to socket 0. + cpus-per-task: "144" + comment: >- + '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"60","reason":"model_loading","description":"Very large model will + take extra loading time."}}' + + srun_options: + mem: "0" + container-remap-root: "" + + telemetry: + collector_join_timeout_seconds: 12 + + benchmark: + placement: + node: head + env: + AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" + AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" + AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" + +override_1p1d: + roles: + prefill: + nodes: 4 + workers: 1 + +override_1p1d_eval: + roles: + prefill: + nodes: 4 + workers: 1 + args: + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + args: + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-variants.yaml new file mode 100644 index 0000000000..de5a7851f9 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3.1/vllm/gb300-fp4/agentx/disagg-variants.yaml @@ -0,0 +1,437 @@ +schema: 2 + +base: + name: kimik3.1-fp4-gb300-dynamo-vllm-agentx-disagg-dcp8 + identity: + model: {} + container: {} + frameworks: + vllm: "0.1.dev24218+g78e9a0c28" + dynamo: "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6" + + # ai-dynamo/dynamo#18124 head: the first Dynamo that registers the kimi_k31 model type. + dynamo: + install: true + source: + rev: "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6" + + # The DSpark draft is read from the shared Hub cache. + environment: + HF_HUB_CACHE: /hf_hub_cache + + slurm: + time_limit: "04:00:00" + + health_check: + max_attempts: 720 + interval_seconds: 10 + + resources: + gpu_type: "gb300" + gpus_per_node: 4 + + services: + - name: etcd + type: etcd + placement: + node: infra + - name: nats + type: nats + placement: + node: infra + options: + max_payload_mb: 32 + - name: mooncake-master + type: mooncake-master + args: + - "--default_kv_lease_ttl=60000" + - "--eviction_high_watermark_ratio=0.95" + - "--eviction_ratio=0.10" + options: + store_config: + metadata_server: "P2PHANDSHAKE" + global_segment_size: "160GB" + local_buffer_size: "4GB" + protocol: "rdma" + device_name: "@fabric.rdma-devices" + mode: "embedded" + enable_offload: false + + frontend: + type: dynamo + enable_multiple_frontends: false + args: + router-mode: "least-loaded" + router-session-affinity-ttl-secs: 1 + env: + DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600" + DYN_TOKENIZER_CACHE_BYTES: "8589934592" + DYN_TCP_REQUEST_TIMEOUT: "120" + DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION: "1" + + engine: + type: vllm + connector: + dp_launch_mode: per_node + + roles: + prefill: + nodes: 2 + workers: 1 + gpus: 8 + env: + <<: &kimi_env + DYN_TCP_REQUEST_TIMEOUT: "120" + VLLM_USE_DIRECT_DCP_A2A: "1" + VLLM_USE_DIRECT_DCP_Q_GATHER: "1" + VLLM_USE_DIRECT_DCP_KV_GATHER: "1" + VLLM_ALLREDUCE_USE_FLASHINFER: "1" + VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT: "0" + DYN_REQUEST_PLANE: "tcp" + ETCD_LEASE_TTL: "600" + VLLM_RPC_TIMEOUT: "600000" + TILELANG_CLEANUP_TEMP_FILES: "1" + VLLM_USE_NCCL_SYMM_MEM: "0" + NCCL_CUMEM_ENABLE: "1" + NCCL_MNNVL_ENABLE: "1" + NCCL_NVLS_ENABLE: "1" + VLLM_SERVER_DEV_MODE: "1" + VLLM_USE_V2_MODEL_RUNNER: "1" + VLLM_USE_RUST_FRONTEND: "1" + VLLM_MOONCAKE_LOAD_RECV_THREADS: "4" + # Lookups lease only the store keys a hit loads. + VLLM_MOONCAKE_STORE_SELECTIVE_LEASE: "1" + MC_SLICE_SIZE: "1048576" + MC_MAX_MR_SIZE: "4294967296" + NCCL_P2P_LEVEL: "NVL" + MC_ENABLE_DEST_DEVICE_AFFINITY: "1" + WITH_NVIDIA_PEERMEM: "0" + NCCL_NET_PLUGIN: "none" + UCX_MEMTYPE_REG_WHOLE: "n" + UCX_TCP_AF_PRIO: "inet" + VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" + # NIXL transfers the KDA conv state in the DS layout. + VLLM_SSM_CONV_STATE_LAYOUT: "DS" + TOKENSPEED_MLA_FP16_PARTIALS: "0" + VLLM_ENGINE_READY_TIMEOUT_S: "3600" + VLLM_ENABLE_GEMM_RS: "1" + DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "1024" + DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "1024" + args: + kv-transfer-config: &kv_transfer '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}' + served-model-name: "moonshotai/Kimi-K3.1-Pro" + prefix-match-unit: 128 + load-format: "fastsafetensors" + kv-cache-dtype: "fp8" + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: "float32" + mamba-cache-dtype: "float32" + gpu-memory-utilization: 0.95 + tensor-parallel-size: 8 + decode-context-parallel-size: 8 + dcp-comm-backend: "a2a" + # NIXL P/D would otherwise raise the DCP interleave after the specialized decode reads it. + cp-kv-cache-interleave-size: 1 + no-enable-flashinfer-autotune: true + enable-cumem-allocator: true + trust-remote-code: true + max-cudagraph-capture-size: 8192 + compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}' + stream-interval: 10 + language-model-only: true + enable-prefix-caching: true + enable-expert-parallel: true + moe-backend: deep_gemm_mega_moe + max-num-seqs: 32 + speculative-config: &spec '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","method":"dspark","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + decode: + nodes: 2 + workers: 1 + gpus: 8 + env: + <<: *kimi_env + VLLM_ENABLE_GEMM_RS: "0" + args: + kv-transfer-config: *kv_transfer + served-model-name: "moonshotai/Kimi-K3.1-Pro" + prefix-match-unit: 128 + load-format: "fastsafetensors" + kv-cache-dtype: "fp8" + attention-config: '{"indexer_kv_dtype":"mxfp4"}' + mamba-ssm-cache-dtype: "float32" + mamba-cache-dtype: "float32" + gpu-memory-utilization: 0.95 + tensor-parallel-size: 8 + decode-context-parallel-size: 8 + dcp-comm-backend: "a2a" + cp-kv-cache-interleave-size: 1 + no-enable-flashinfer-autotune: true + enable-cumem-allocator: true + trust-remote-code: true + stream-interval: 10 + language-model-only: true + enable-prefix-caching: true + additional-config: '{"decode_backend":"latency_specialized"}' + use-replayssm: true + max-num-seqs: 64 + max-cudagraph-capture-size: 320 + compilation-config: '{"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320]}' + speculative-config: *spec + + sbatch_directives: + mem: "0" + # 72 pins every vLLM worker to socket 0. + cpus-per-task: "144" + comment: >- + '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"60","reason":"model_loading","description":"Very large model will + take extra loading time."}}' + + srun_options: + mem: "0" + container-remap-root: "" + + telemetry: + collector_join_timeout_seconds: 12 + + benchmark: + placement: + node: head + env: + AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" + AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" + AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" + +override_1p1d: + roles: + prefill: &override_1p1d_prefill + nodes: 2 + workers: 1 + args: &override_1p1d_prefill_args {} + decode: &override_1p1d_decode + nodes: 2 + workers: 1 + args: &override_1p1d_decode_args {} + +override_1p1d_eval: + roles: + prefill: + <<: *override_1p1d_prefill + args: + <<: *override_1p1d_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_1p1d_decode + args: + <<: *override_1p1d_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + +override_1p2d: + roles: + prefill: &override_1p2d_prefill + nodes: 2 + workers: 1 + args: &override_1p2d_prefill_args + numa-bind: true + numa-bind-nodes: [0, 0, 1, 1] + decode: &override_1p2d_decode + nodes: 4 + workers: 2 + args: &override_1p2d_decode_args + numa-bind: true + numa-bind-nodes: [0, 0, 1, 1] + +override_1p2d_eval: + roles: + prefill: + <<: *override_1p2d_prefill + args: + <<: *override_1p2d_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_1p2d_decode + args: + <<: *override_1p2d_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + +override_1p3d: + roles: + prefill: &override_1p3d_prefill + nodes: 2 + workers: 1 + args: &override_1p3d_prefill_args + numa-bind: true + numa-bind-nodes: [0, 0, 1, 1] + decode: &override_1p3d_decode + nodes: 6 + workers: 3 + args: &override_1p3d_decode_args + numa-bind: true + numa-bind-nodes: [0, 0, 1, 1] + +override_1p3d_eval: + roles: + prefill: + <<: *override_1p3d_prefill + args: + <<: *override_1p3d_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_1p3d_decode + args: + <<: *override_1p3d_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + +# 256 decode slots per TP8 engine for the higher-concurrency points. +override_1p2d_mns256: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: &override_1p2d_mns256_prefill + nodes: 2 + workers: 1 + env: &override_mns256_prefill_env + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + args: &override_1p2d_mns256_prefill_args {} + decode: &override_1p2d_mns256_decode + nodes: 4 + workers: 2 + env: &override_mns256_decode_env + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + args: &override_1p2d_mns256_decode_args + max-num-seqs: 256 + max-num-batched-tokens: 1280 + max-cudagraph-capture-size: 1280 + compilation-config: &decode_graphs_1280 '{"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320,340,360,380,400,420,440,460,480,500,520,540,560,580,600,620,640,660,680,700,720,740,760,780,800,820,840,860,880,900,920,940,960,980,1000,1020,1040,1060,1080,1100,1120,1140,1160,1180,1200,1220,1240,1260,1280]}' + +override_1p2d_mns256_eval: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: + <<: *override_1p2d_mns256_prefill + args: + <<: *override_1p2d_mns256_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_1p2d_mns256_decode + args: + <<: *override_1p2d_mns256_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + +override_1p3d_mns256: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: &override_1p3d_mns256_prefill + nodes: 2 + workers: 1 + env: *override_mns256_prefill_env + args: &override_1p3d_mns256_prefill_args {} + decode: &override_1p3d_mns256_decode + nodes: 6 + workers: 3 + env: *override_mns256_decode_env + args: &override_1p3d_mns256_decode_args + max-num-seqs: 256 + max-num-batched-tokens: 1280 + max-cudagraph-capture-size: 1280 + compilation-config: *decode_graphs_1280 + +override_1p3d_mns256_eval: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: + <<: *override_1p3d_mns256_prefill + args: + <<: *override_1p3d_mns256_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_1p3d_mns256_decode + args: + <<: *override_1p3d_mns256_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + +# Hybrid decode: four TP8 x DCP8 engines in DP, MegaMoE EP32 across 32 GPUs. +override_hyb4_3p1d: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: &override_hyb4_3p1d_prefill + nodes: 6 + workers: 3 + env: *override_mns256_prefill_env + args: &override_hyb4_3p1d_prefill_args + max-num-batched-tokens: 16384 + decode: &override_hyb4_3p1d_decode + nodes: 8 + workers: 1 + gpus: 32 + env: *override_mns256_decode_env + args: &override_hyb4_3p1d_decode_args + data-parallel-size: 4 + data-parallel-rpc-port: 13345 + enable-expert-parallel: true + moe-backend: deep_gemm_mega_moe + gpu-memory-utilization: 0.92 + max-num-seqs: 256 + max-num-batched-tokens: 1280 + max-cudagraph-capture-size: 1280 + compilation-config: *decode_graphs_1280 + +override_hyb4_3p1d_eval: + engine: + # Lets vLLM bind each spawned worker to its GPU's NUMA node. + set_visible_devices: true + roles: + prefill: + <<: *override_hyb4_3p1d_prefill + args: + <<: *override_hyb4_3p1d_prefill_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true + decode: + <<: *override_hyb4_3p1d_decode + args: + <<: *override_hyb4_3p1d_decode_args + reasoning-parser: kimi_k3 + dyn-tool-call-parser: kimi_k3 + dyn-reasoning-parser: kimi_k3 + dyn-enable-structural-tag: true diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index fe4f5aad00..86559e660c 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -8005,6 +8005,103 @@ kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk: kv-offloading: none worker: { num-worker: 1, tp: 16, ep: 1, dp-attn: false } +kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg: + image: vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9cb47f6215add0cba70b4b7d3596e6f3990d8b46c7cad0f8fbbc8b + model: moonshotai/Kimi-K3.1-Pro + model-prefix: kimik3.1 + runner: cluster:gb300-nv + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: 3dd0744eb72c707d76d4c520ae4dea9e286a4ab6 } + kv-p2p-transfer: nixl + multinode: true + disagg: true + srt-recipe-dir: kimik3.1/vllm/gb300-fp4/agentx + scenarios: + agentic-coding: + - dram-utilization: 0.1775 + search-space: + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [1] + srt-recipe: disagg-variants.yaml:override_1p1d + eval-srt-recipe: disagg-variants.yaml:override_1p1d_eval + power: true + prefill: { num-worker: 1, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 1, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [20, 32] + srt-recipe: disagg-variants.yaml:override_1p3d + eval-srt-recipe: disagg-variants.yaml:override_1p3d_eval + power: true + prefill: { num-worker: 1, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [56] + srt-recipe: disagg-variants.yaml:override_1p3d_mns256 + eval-srt-recipe: disagg-variants.yaml:override_1p3d_mns256_eval + power: true + prefill: { num-worker: 1, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 3, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [48] + srt-recipe: disagg-variants.yaml:override_1p2d + eval-srt-recipe: disagg-variants.yaml:override_1p2d_eval + power: true + prefill: { num-worker: 1, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 2, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [64, 80] + srt-recipe: disagg-variants.yaml:override_1p2d_mns256 + eval-srt-recipe: disagg-variants.yaml:override_1p2d_mns256_eval + power: true + prefill: { num-worker: 1, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 2, tp: 8, dcp-size: 8, ep: 1, dp-attn: false } + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [384] + srt-recipe: disagg-variants.yaml:override_hyb4_3p1d + eval-srt-recipe: disagg-variants.yaml:override_hyb4_3p1d_eval + power: true + prefill: { num-worker: 3, tp: 8, dcp-size: 8, ep: 8, dp-attn: false } + decode: { num-worker: 1, tp: 32, dcp-size: 8, ep: 32, dp-attn: true } + +kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg: + image: vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9cb47f6215add0cba70b4b7d3596e6f3990d8b46c7cad0f8fbbc8b + model: moonshotai/Kimi-K3.1-Pro + model-prefix: kimik3.1 + runner: cluster:gb300-nv + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: 3dd0744eb72c707d76d4c520ae4dea9e286a4ab6 } + kv-p2p-transfer: nixl + multinode: true + disagg: true + srt-recipe-dir: kimik3.1/vllm/gb300-fp4/agentx + scenarios: + agentic-coding: + - dram-utilization: 0.1775 + search-space: + - spec-decoding: mtp + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: 0.3.14 } + conc-list: [300, 352] + srt-recipe: disagg-dep16.yaml:override_1p1d + eval-srt-recipe: disagg-dep16.yaml:override_1p1d_eval + power: true + prefill: { num-worker: 1, tp: 16, ep: 16, dp-attn: true } + decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: true } + qwen3.8next-fp4-gb300-tokenspeed-agentic-mtp: image: ghcr.io/lightseekorg/tokenspeed-runner:cu130-torch-2.14.0-flashinfer-0.7.1rc2@sha256:1086943cd27a22b7d414c64ca088bee24f53a971fdae78e231c182c50b4e43d2 model: nvidia/Qwen3.8-Flash-Next-NVFP4 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 86f1786a8b..830b3bf619 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12115,6 +12115,15 @@ - "Aggregated low-latency points: one TP16 EP1 worker with the NVIDIA megakernel (--enable-nvidia-megakernel), max-running-requests equal to the concurrency (c1, c2, c4)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811 +- config-keys: + - kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg + - kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg + scenario-type: + - agentic-coding + description: + - "Add Kimi-K3.1-Pro GB300 Dynamo+vLLM disaggregated AgentX configs." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4839 + - config-keys: - qwen3.8next-fp4-gb300-tokenspeed-agentic-mtp scenario-type: From 37cf0dfc4648eb88e8f185950583cf234837ee6c Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:45:57 -0800 Subject: [PATCH 014/111] =?UTF-8?q?[VideoGenX]=20sweep=20H3=20latency=20on?= =?UTF-8?q?=201,=202,=204=20and=208=20H200=20GPUs=20/=20[VideoGenX]=20?= =?UTF-8?q?=E5=9C=A8=201=E3=80=812=E3=80=814=E3=80=818=20=E5=BC=A0=20H200?= =?UTF-8?q?=20=E4=B8=8A=E6=89=AB=E6=8F=8F=20H3=20=E5=BB=B6=E8=BF=9F=20(#48?= =?UTF-8?q?56)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: sweep H3 video on 1, 2, 4 and 8 H200 GPUs * chore: link H200 GPU-count changelog entry to #4856 --- .../h3/sglang/h200-bf16/serving-client.yaml | 65 +++++++++++++++++++ inferencex-e2e/configs/nvidia-master.yaml | 27 ++++++++ inferencex-e2e/docs/MODELS.md | 2 +- inferencex-e2e/docs/video-generation.md | 8 +-- inferencex-e2e/perf-changelog.yaml | 7 ++ video-generation/README.md | 2 +- video-generation/README_zh.md | 2 +- 7 files changed, 106 insertions(+), 7 deletions(-) create mode 100644 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h200-bf16/serving-client.yaml diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h200-bf16/serving-client.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h200-bf16/serving-client.yaml new file mode 100644 index 0000000000..c5ebf27717 --- /dev/null +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/h200-bf16/serving-client.yaml @@ -0,0 +1,65 @@ +schema: 2 +name: h3-sglang-serving-client + +resources: + gpu_type: h200 + +services: + - name: video-server + type: generic + nodes: 1 + critical: true + start: before_workers + command: [python3, -m, infx.bench, video, serve] + args: + - --model-view + - snapshot + - -- + - /opt/sglang/bin/python3 + - -m + - evaluator.srt_selected + - serve + - --model-type + - diffusion + - --model-path + - "{model_path}" + - --model-id + - "{model_id}" + - --revision + - "{model_revision}" + - --model-variant + - fl2va + - --num-gpus + - "{gpus_per_node}" + - --ulysses-degree + - "{gpus_per_node}" + - --tp-size + - "1" + - --encoder-parallel + - auto + - --warmup-resolutions + - 1344x768 + - --disable-conditioning-cache + - --performance-mode + - speed + - --host + - 0.0.0.0 + - --port + - "30283" + - --enable-torch-compile + - "false" + - --dit-cpu-offload + - "false" + env: + PYTHONPATH: /video-package:/infmax-workspace + PYTHONUNBUFFERED: "1" + VIDEO_SERVER_PORT: "30283" + VIDEO_RUNTIME_REVISION: 297e54fd105e974cdf401e5b4d20253a22c16f90 + HF_HUB_OFFLINE: "1" + TRANSFORMERS_OFFLINE: "1" + readiness: + http: + port: 30283 + path: /health + timeout_seconds: 1800 + interval_seconds: 5 diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 86559e660c..e780521744 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7489,6 +7489,33 @@ h3-bf16-h100-sglang-video: srt-recipe: serving-client-tp2-ulysses4.yaml conc-list: [1] +h3-bf16-h200-sglang-video: + image: lmsysorg/sglang:v0.5.23@sha256:f7876bef696ccf9cf75a2bf7ccb82970f5fac577f082fb8ca95c9a59d2160eb5 + model: MiniMaxAI/MiniMax-H3 + model-prefix: h3 + runner: cluster:h200-dgxc + precision: bf16 + framework: sglang + multinode: false + srt-recipe-dir: h3/sglang/h200-bf16 + scenarios: + video-generation: + - spec: h3-v2.yaml + set: release-v1 + search-space: + - gpus: 1 + srt-recipe: serving-client.yaml + conc-list: [1] + - gpus: 2 + srt-recipe: serving-client.yaml + conc-list: [1] + - gpus: 4 + srt-recipe: serving-client.yaml + conc-list: [1] + - gpus: 8 + srt-recipe: serving-client.yaml + conc-list: [1] + h3-bf16-b200-sglang-video: image: lmsysorg/sglang:v0.5.23@sha256:f7876bef696ccf9cf75a2bf7ccb82970f5fac577f082fb8ca95c9a59d2160eb5 model: MiniMaxAI/MiniMax-H3 diff --git a/inferencex-e2e/docs/MODELS.md b/inferencex-e2e/docs/MODELS.md index 2255d5898c..5433970748 100644 --- a/inferencex-e2e/docs/MODELS.md +++ b/inferencex-e2e/docs/MODELS.md @@ -36,7 +36,7 @@ For non-grandfathered models, `ROCM_QUICK_REDUCE_QUANTIZATION`, `VLLM_ROCM_QUICK |---|---|---| | Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see the srt-slurm recipes under [`benchmarks/single_node/srt-slurm-recipes/`](../benchmarks/single_node/srt-slurm-recipes) and the shared client [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation policy](#deprecation-policy)). | | Single-turn 8k1k | 8192 / 1024 | Active. This is the primary fixed-sequence-length scenario. | -| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 GPUs, 2, 4 and 8 H100 GPUs and 4 MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | +| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 or H200 GPUs, 2, 4 and 8 H100 GPUs and 4 MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | | Single-turn 1k1k | 1024 / 1024 | Deprecated since 2026-07-17 ([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Archived configs lived in `configs/deprecated/` until it was deleted (#3463); see git history. The GLM-5.1 B200 TileRT point added later in [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) was the last 1k1k config; it was retired on 2026-10-01 ([#3638](https://github.com/SemiAnalysisAI/InferenceX/pull/3638)). | | Single-turn 1k8k | 1024 / 8192 | **Deprecated for all models** since 2026-03-27 ([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Configs were removed, not archived. | diff --git a/inferencex-e2e/docs/video-generation.md b/inferencex-e2e/docs/video-generation.md index 4b51acd3b8..e954998ab8 100644 --- a/inferencex-e2e/docs/video-generation.md +++ b/inferencex-e2e/docs/video-generation.md @@ -1,9 +1,9 @@ # Native video delivery The `video-generation` scenario schedules one server job per deployment. SGLang -selects C1 deployments on 1, 2, 4 and 8 B200 GPUs (Ulysses across all GPUs), on -2, 4 and 8 H100 GPUs (TP2, since the model does not fit one 80 GB GPU) and on 4 -MI355X GPUs, all using +selects C1 deployments on 1, 2, 4 and 8 B200 or H200 GPUs (Ulysses across all +GPUs), on 2, 4 and 8 H100 GPUs (TP2, since the model does not fit one 80 GB GPU) +and on 4 MI355X GPUs, all using `h3-v2.yaml`: 20 distinct measured prompts, a separate warmup, 49 denoising updates, and 124 frames at 24 fps and 1344×768. A five-second request produces 124/24 seconds of video. Config presence is not GPU or quality qualification. @@ -112,7 +112,7 @@ Their presence does not establish final-head GPU qualification. 2. Change the existing row in its master config, or copy it with a distinct key and recipe for a deployment that must remain separately selectable. Pin the image by digest and use its real interpreter and architecture. The - H100/B200 SGLang rows use the public CUDA amd64 image and the MI355X row its + H100/H200/B200 SGLang rows use the public CUDA amd64 image and the MI355X row its ROCm MI35x build. A shared image does not qualify a different topology or SKU: validate the selected deployments through their sweep. Maintainers stage model snapshots in the runner's diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 830b3bf619..4fe1531967 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12338,3 +12338,10 @@ - "Update the vLLM image from nightly-dev-arm64-cu130-ac9126e58aa7 to the digest-pinned nightly-00b6c4c5." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4558 scenario-type: [agentic-coding] + +- config-keys: + - h3-bf16-h200-sglang-video + scenario-type: [video-generation] + description: + - "Restore the H200 row and sweep H3 C1 latency on 1, 2, 4 and 8 H200 GPUs with SGLang Ulysses across all GPUs." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4856 diff --git a/video-generation/README.md b/video-generation/README.md index e85eea9070..e96dd17a55 100644 --- a/video-generation/README.md +++ b/video-generation/README.md @@ -15,7 +15,7 @@ them from `perf-changelog.yaml`, `run-sweep.yml` runs the `video` bucket through the SKU recipe under `benchmarks/single_node/srt-slurm-recipes/h3/` starts the engine through `python3 -m infx.bench video serve` as a generic service and runs `python3 -m infx.bench video client` as `benchmark.type: custom`. This directory -is mounted into the job as `/video-package`. H100, B200 and MI355X select +is mounted into the job as `/video-package`. H100, H200, B200 and MI355X select `h3-v2.yaml/release-v1`: 20 distinct measured prompts at C1. See [native video delivery](../inferencex-e2e/docs/video-generation.md) for artifact identity, reuse validation and post-merge publication. diff --git a/video-generation/README_zh.md b/video-generation/README_zh.md index 97282aa2d2..1e12bbefd4 100644 --- a/video-generation/README_zh.md +++ b/video-generation/README_zh.md @@ -13,7 +13,7 @@ diffusion 引擎在单节点上提供服务，选定的冻结 YAML 工作负载 `benchmarks/single_node/srt-slurm-recipes/h3/` 下的 SKU 配方通过 `python3 -m infx.bench video serve` 把引擎作为 generic service 启动， 并以 `benchmark.type: custom` 运行 `python3 -m infx.bench video client`。 -本目录以 `/video-package` 挂载进任务。H100、B200 与 MI355X 选用 +本目录以 `/video-package` 挂载进任务。H100、H200、B200 与 MI355X 选用 `h3-v2.yaml/release-v1`，在 C1 测量 20 个不同 prompt。 产物身份、复用校验与合并后发布见 [原生视频交付](../inferencex-e2e/docs/video-generation.md)。 From 264df9ccf3b252da4b4567d0274b7bbd77f728bc Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Fri, 6 Nov 2026 12:19:59 -0800 Subject: [PATCH 015/111] =?UTF-8?q?[VideoGenX]=20sweep=20H3=20latency=20on?= =?UTF-8?q?=201,=202,=204=20and=208=20MI355X=20GPUs=20/=20[VideoGenX]=20?= =?UTF-8?q?=E5=9C=A8=201=E3=80=812=E3=80=814=E3=80=818=20=E5=BC=A0=20MI355?= =?UTF-8?q?X=20=E4=B8=8A=E6=89=AB=E6=8F=8F=20H3=20=E5=BB=B6=E8=BF=9F=20(#4?= =?UTF-8?q?818)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: run native H3 video sweeps from YAML recipes * feat: verify GPU ownership and seal H3 execution evidence * feat: define reproducible H3 workload sets * feat: measure H3 requests and validate generated media * refactor: delete unused OwnedProcess.owns helper * refactor: delete unused H3 bundle JSON digest helper * test: restore shared launch and workflow test cases * refactor: fold H3 spec digest and case helpers * test: table-drive H3 spec rejection cases * chore: compact H3 spec YAML and drop stale references * fix: use warmup case in GPU report slots * refactor: drop unreachable GPU job guards * test: drop dead parametrize branches in measurement tests * refactor: share no-follow open and canonical JSON helpers * test: inline single-case video parametrize and share reseal fixtures * refactor: share deadline budget in GPU probes * refactor: reuse runner slots and hashing in compare * test: drop dead parametrize branches in GPU tests * refactor: dedupe evaluator encoding, status, and quantile rules * refactor: drop duplicated video digest, srun, schema and env helpers * refactor: share video srt recipe defaults like agentic * docs: link video pins and timeouts to their sources * refactor: move offline compare and report tools out of measurement core * refactor: move offline GPU job tools out of native H3 sweep * refactor: move manual srt-yaml commands out of native H3 sweep * style: format native H3 evaluator modules with ruff * refactor: expose canonical_json_bytes as the one H3 hash encoding * fix: check the video readiness port and drop the empty prepare description * refactor: drop citations and per-engine notes from H3 specs * test: drop the hand-built fairness hash test * refactor: validate H3 v0.1 policy once in the spec loader by schema version * ci: lint and format all of experimental/video-generation and drop measurement-only deps * revert: keep run comparison and PSNR in the measurement core * refactor: move SGLang server timing out of the measurement core * chore: drop the unreferenced smoke plan and the nonexistent mvp extra hint * style: ruff format the measurement core * refactor: share the native video file contract from infx.bench.video * refactor: hash run bundles with the spec's canonical JSON * refactor: name each rejected field of the sealed v0.1 server launch * refactor: trust loader-validated plans and drop the redundant run_plan model_revision * refactor: derive engine encoding requests and media checks from the plan's encoding * refactor: keep native video checks at the binder, executor and ingest boundaries * chore: drop unconsumed prose and constant fields from run data * fix: keep a bounded redacted HTTP error excerpt and refuse every paid MiniMax host * refactor: default the single-choice reference dtype and attention backend * refactor: record native video job status as JSON * fix: count malformed power samples in the video bundle instead of skipping them * fix: drop unread constant bundle fields and validate each power sample's GPU coverage * refactor: resolve the video harness at one path beside the project * fix: hash the measurement window run with the shared canonical JSON * refactor: seal the image digest for every native video plan and check the launch once * fix: keep the video workload specs in nested matrix snapshots * fix: report the native video bundle outcome as its CI phase * fix: count video sweep jobs in run stats after they finish * fix: drop the static protocol source pin so run configuration names only the image revision * refactor: rename generic H3 spec and workflow names to video * refactor: rename generic H3 measurement-core naming to video * refactor: rename generic H3 native-sweep naming to video * fix: record approved head and video matrix in trusted sweeps * fix: validate trusted video sweeps with the dispatching workflow's validator * feat: run H3 on SGLang v0.5.23 for B200 and H100 (#4776) * fix: record engine parallelism in native video spec.json (#4777) * chore: fold the v0.5.23 changelog entry into the H3 sweep entry * chore: drop the H200 vLLM-Omni cell from the H3 sweep * chore: run H3 H200 on SGLang v0.5.23 * chore: drop the H200 SGLang cell from the H3 sweep * refactor: move the video harness to top-level video-generation * refactor: name the 20-prompt H3 spec by its schema version * docs: drop the release label from the runner's policy note * fix: let vgbench run record the server's numerical identity * fix: check reference case IDs before selecting all cases * docs: drop the release label from the H3 docs and changelog entry * fix: accept the newest video bundle per deployment across rerun attempts * fix: retry the video results upload like other result uploads * fix: stop serving-mode video runs promptly on interrupt * fix: name ROCm video GPUs by board serial * feat: add MiniMax-H3 MI355X SGLang video sweep * feat: serve MI355X H3 video from the shared HF cache * docs: link the MI355X H3 changelog entry to its PR * fix: select AITER attention in the MI355X H3 recipe * feat: sweep H3 video latency across 1, 2, 4 and 8 GPUs * fix: keep a single video bundle in its named directory [skip-sweep] * chore: link GPU-count changelog entry to #4818 [skip-sweep] * ci: raise video job limit to 150 minutes for 1-GPU points * refactor: share single-node srt steps between points and video [skip-sweep] * feat: sweep B200 H3 video across 1, 2, 4 and 8 GPUs * chore: move B200 GPU-count points to #4852 * chore: narrow the GPU-count sweep to MI355X --- .../h3/sglang/mi355x-bf16/serving-client.yaml | 2 +- inferencex-e2e/configs/amd-master.yaml | 9 +++++++++ inferencex-e2e/docs/MODELS.md | 2 +- inferencex-e2e/docs/video-generation.md | 6 +++--- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 5 files changed, 21 insertions(+), 5 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/mi355x-bf16/serving-client.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/mi355x-bf16/serving-client.yaml index a98cdd78bf..1c38403fca 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/mi355x-bf16/serving-client.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/sglang/mi355x-bf16/serving-client.yaml @@ -32,7 +32,7 @@ services: - --num-gpus - "{gpus_per_node}" - --ulysses-degree - - "4" + - "{gpus_per_node}" - --tp-size - "1" - --attention-backend diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 07cf9604ba..e8c34a2cda 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -1123,6 +1123,15 @@ h3-bf16-mi355x-sglang-video: - spec: h3-v2.yaml set: release-v1 search-space: + - gpus: 1 + srt-recipe: serving-client.yaml + conc-list: [1] + - gpus: 2 + srt-recipe: serving-client.yaml + conc-list: [1] - gpus: 4 srt-recipe: serving-client.yaml conc-list: [1] + - gpus: 8 + srt-recipe: serving-client.yaml + conc-list: [1] diff --git a/inferencex-e2e/docs/MODELS.md b/inferencex-e2e/docs/MODELS.md index 5433970748..a5578e53ec 100644 --- a/inferencex-e2e/docs/MODELS.md +++ b/inferencex-e2e/docs/MODELS.md @@ -36,7 +36,7 @@ For non-grandfathered models, `ROCM_QUICK_REDUCE_QUANTIZATION`, `VLLM_ROCM_QUICK |---|---|---| | Agentic coding | Long Context, Multi Turn Realistic traffic trace replay with sub agents | Active. This is the trace-replay agentic-coding benchmark (see the srt-slurm recipes under [`benchmarks/single_node/srt-slurm-recipes/`](../benchmarks/single_node/srt-slurm-recipes) and the shared client [`benchmarks/srt_agentic.sh`](../benchmarks/srt_agentic.sh)). Going forward, new models will likely be onboarded with agentic coding only. Speculative decoding may be enabled or disabled to produce the best Pareto points; a separate non-spec-decode A/B baseline is not required (see [Deprecation policy](#deprecation-policy)). | | Single-turn 8k1k | 8192 / 1024 | Active. This is the primary fixed-sequence-length scenario. | -| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200 or H200 GPUs, 2, 4 and 8 H100 GPUs and 4 MI355X GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | +| Video generation | H3: 124 frames at 24 fps, 1344x768, C1 | MiniMax-H3 uses native srt-slurm: SGLang selects 20-clip `h3-v2.yaml` C1 points on 1, 2, 4 and 8 B200, H200 or MI355X GPUs and 2, 4 and 8 H100 GPUs. The client and server live in [`video-generation`](../../video-generation); energy is reported per valid clip. Config presence does not qualify execution, power, or quality. See [video delivery](video-generation.md). | | Single-turn 1k1k | 1024 / 1024 | Deprecated since 2026-07-17 ([#2263](https://github.com/SemiAnalysisAI/InferenceX/pull/2263)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Archived configs lived in `configs/deprecated/` until it was deleted (#3463); see git history. The GLM-5.1 B200 TileRT point added later in [#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533) was the last 1k1k config; it was retired on 2026-10-01 ([#3638](https://github.com/SemiAnalysisAI/InferenceX/pull/3638)). | | Single-turn 1k8k | 1024 / 8192 | **Deprecated for all models** since 2026-03-27 ([#911](https://github.com/SemiAnalysisAI/InferenceX/pull/911)), to save GPU cluster time for higher-priority real-world agentic-coding benchmarks and new frontier models. Configs were removed, not archived. | diff --git a/inferencex-e2e/docs/video-generation.md b/inferencex-e2e/docs/video-generation.md index e954998ab8..604d76b55f 100644 --- a/inferencex-e2e/docs/video-generation.md +++ b/inferencex-e2e/docs/video-generation.md @@ -1,9 +1,9 @@ # Native video delivery The `video-generation` scenario schedules one server job per deployment. SGLang -selects C1 deployments on 1, 2, 4 and 8 B200 or H200 GPUs (Ulysses across all -GPUs), on 2, 4 and 8 H100 GPUs (TP2, since the model does not fit one 80 GB GPU) -and on 4 MI355X GPUs, all using +selects C1 deployments on 1, 2, 4 and 8 B200, H200 or MI355X GPUs (Ulysses +across all GPUs) and on 2, 4 and 8 H100 GPUs (TP2, since the model does not fit +one 80 GB GPU), all using `h3-v2.yaml`: 20 distinct measured prompts, a separate warmup, 49 denoising updates, and 124 frames at 24 fps and 1344×768. A five-second request produces 124/24 seconds of video. Config presence is not GPU or quality qualification. diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 4fe1531967..1e8edb5c0b 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12345,3 +12345,10 @@ description: - "Restore the H200 row and sweep H3 C1 latency on 1, 2, 4 and 8 H200 GPUs with SGLang Ulysses across all GPUs." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4856 + +- config-keys: + - h3-bf16-mi355x-sglang-video + scenario-type: [video-generation] + description: + - "Sweep H3 C1 latency on 1, 2, 4 and 8 MI355X GPUs with SGLang Ulysses across all GPUs; the MI355X recipe now sets the Ulysses degree from the GPU count, which keeps the 4-GPU point unchanged." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4818 From 610e59e9f2d6c7c9dd8c98d4031a6897318399fd Mon Sep 17 00:00:00 2001 From: hshrivastava-droid Date: Fri, 6 Nov 2026 13:06:59 -0800 Subject: [PATCH 016/111] =?UTF-8?q?[NV]=20Qwen3.8-Flash-Next=20FP4=20GB300?= =?UTF-8?q?=20SGLang=20AgentX=20update=20/=20[NV]=20Qwen3.8-Flash-Next=20F?= =?UTF-8?q?P4=20GB300=20SGLang=20AgentX=20=E6=9B=B4=E6=96=B0=20(#4787)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [NV] Qwen3.8-Flash-Next FP4 GB300 SGLang AgentX update Signed-off-by: hshrivastava-droid * Update perf-changelog pr-link Signed-off-by: hshrivastava-droid * Update perf-changelog description Signed-off-by: hshrivastava-droid --------- Signed-off-by: hshrivastava-droid Co-authored-by: Claude Sonnet 5.5 --- .../sglang/gb300-fp4-mtp/agentic.yaml | 175 +++++++++++++++--- inferencex-e2e/configs/nvidia-master.yaml | 18 +- inferencex-e2e/perf-changelog.yaml | 9 + 3 files changed, 167 insertions(+), 35 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/gb300-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/gb300-fp4-mtp/agentic.yaml index 119a4e953e..f4f4f40281 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/gb300-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/gb300-fp4-mtp/agentic.yaml @@ -29,6 +29,9 @@ base: SGLANG_TIMEOUT_KEEP_ALIVE: '1800' # Explicitly disable the prohibited draft precision conversion path. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '0' + # NVIDIA runtime optimizations: compact verify sampling (verify tokens sampled on the top-k support before + # temperature and top-p; same distribution), post-verify CUDA graphs and their fast paths. + SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION: '1' args: served-model-name: nvidia/Qwen3.8-Flash-Next-NVFP4 revision: fc694b54fb0174e0913e6adf86691ef85a4ead47 @@ -64,7 +67,7 @@ base: # The native connector injects model-specific golden AL for throughput; # quality evaluations use real verification. No manual AL override. enable-linear-replayssm-spec: true - cuda-graph-backend-prefill: disabled + cuda-graph-backend-prefill: breakable cuda-graph-backend-decode: full cuda-graph-max-bs-decode: 96 cuda-graph-bs-decode: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96] @@ -101,43 +104,171 @@ base: TOTAL_CPU_DRAM_GB: '241' KV_OFFLOADING: dram -# Only client concurrency changes; serving/admission/graph/cache settings stay fixed. -override_c1: - benchmark: - env: - CONC: '1' - -override_c2: - benchmark: - env: - CONC: '2' - -override_c4: +override_tp4_c3: + roles: + agg: + gpus: 4 + env: + # Captures the decode step's host-side metadata preparation in the decode CUDA graph. + SGLANG_ENABLE_METADATA_GLUE_GRAPH: '1' + args: + <<: &gpu_only_args + # null drops the base setting, so SGLang's default applies. + enable-hierarchical-cache: null + hicache-size: null + hicache-io-backend: null + hicache-mem-layout: null + hicache-write-policy: null + mamba-radix-cache-strategy: null + mamba-track-interval: null + max-mamba-cache-size: null + # Dense small sizes: decode batches up to 16 pad by at most one request. + cuda-graph-bs-decode: [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96] + tokenizer-worker-num: 6 + tensor-parallel-size: 4 + expert-parallel-size: 4 + # SGLang's custom all-reduce instead of the FlashInfer fused all-reduce. + enforce-disable-flashinfer-allreduce-fusion: true + # Keep the per-layer embedding table in GPU memory instead of pinned host memory. + ple-offload-embedding: null + ple-offload-backend: null + no-ple-offload-embedding: true + # Running cap: one request decodes at a time; the other sessions wait for their first token. + max-running-requests: 1 benchmark: env: - CONC: '4' + <<: &gpu_only_client + KV_OFFLOADING: none + TOTAL_CPU_DRAM_GB: '0' + CONC: '3' override_c8: + roles: + agg: + args: + <<: *gpu_only_args + # Running cap with shortest-prefill-first admission: two requests decode; short new requests start first. + max-running-requests: 2 + schedule-policy: shortest-prefill-first benchmark: env: + <<: *gpu_only_client CONC: '8' -override_c16: +override_c20: + roles: + agg: + env: + # Cache-aware admission skips its in-batch prefix check, which holds short new requests behind long ones. + IN_BATCH_PREFIX_CACHING_CHECK_THRESHOLD: '-1' + # One fused MoE routing kernel for 17+ token batches instead of the split path; outputs are bitwise equal. + FLASHINFER_ROUTING_FORCE_BLOCK_PER_TOKEN: '0' + args: + <<: *gpu_only_args + kv-cache-dtype: fp8_e4m3 + speculative-draft-kv-cache-dtype: fp8_e4m3 + # Context length: no decode-time Mamba checkpoints; the AgentX replay never re-sends model output. + mamba-track-interval: 262144 + cuda-graph-bs-decode: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96] + # Running cap with shortest-prefill-first admission: short new requests start first; long ones wait. + max-running-requests: 6 + schedule-policy: shortest-prefill-first + # Hold new prefills until two running slots are free, so they share one prefill pass. + min-free-slots-delay: 2 + # FP8 QSA indexer: the sparse-attention indexer keeps its keys in FP8 and scores with FP8 GEMMs. + qsa-indexer-dtype: fp8_e4m3 benchmark: env: - CONC: '16' + <<: *gpu_only_client + CONC: '20' -override_c32: +override_c36: + roles: + agg: + env: + <<: &hicache_env + # The switch's post-verify CUDA graphs and their fast paths stay off here: capturing them lazily while + # serving ran out of GPU memory at concurrency 80, and they gave no gain at 36 and 56. + # Compact verify sampling stays on. + SGLANG_ENABLE_EAGLE_POST_VERIFY_CUDA_GRAPH: '0' + SGLANG_OPT_SPEC_VERIFY_REUSE_STAGED_INPUTS: '0' + SGLANG_OPT_SPEC_POST_VERIFY_ONE_LAUNCH_LOAD: '0' + SGLANG_OPT_SMALL_FIELD_COPY_VECTORIZED: '0' + SGLANG_OPT_SPEC_PACKED_VERIFY_BCAST: '0' + SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH: '0' + IN_BATCH_PREFIX_CACHING_CHECK_THRESHOLD: '-1' + FLASHINFER_ROUTING_FORCE_BLOCK_PER_TOKEN: '0' + args: + <<: &hicache_fp8_kv_args + kv-cache-dtype: fp8_e4m3 + speculative-draft-kv-cache-dtype: fp8_e4m3 + # Context length: no decode-time Mamba checkpoints; the AgentX replay never re-sends model output. + mamba-track-interval: 262144 + min-free-slots-delay: 2 + # Two draft steps (three draft tokens): a smaller verify batch per decode step at these running sizes. + speculative-num-steps: 2 + speculative-num-draft-tokens: 3 + max-mamba-cache-size: 720 + cuda-graph-bs-decode: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 56, 64, 72, 80, 88, 96] + max-running-requests: 12 + schedule-policy: shortest-prefill-first benchmark: env: - CONC: '32' + CONC: '36' -override_c48: +override_c56: + roles: + agg: + env: *hicache_env + args: + <<: *hicache_fp8_kv_args + max-mamba-cache-size: 720 + cuda-graph-bs-decode: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 60, 64, 72, 80, 88, 96] + # Running cap: at most 35 requests decode; the rest wait for their first token. + max-running-requests: 35 + # Hold prefill until the queue reaches min(20% of running, 4) requests or 300 ms pass; at most four per pass. + enable-prefill-delayer: true + prefill-delayer-queue-min-ratio: 0.2 + prefill-delayer-max-delay-ms: 300 + prefill-max-requests: 4 + # After a prefill pass, run ten decode rounds before the next while 12 or more requests are running. + prefill-decode-interval: 10 + prefill-decode-interval-min-bs: 12 + # FP8 QSA indexer: the sparse-attention indexer keeps its keys in FP8 and scores with FP8 GEMMs. + qsa-indexer-dtype: fp8_e4m3 benchmark: env: - CONC: '48' + CONC: '56' -override_c64: +override_c80: + roles: + agg: + env: *hicache_env + args: + <<: *hicache_fp8_kv_args + speculative-num-steps: 2 + speculative-num-draft-tokens: 3 + mem-fraction-static: 0.90 + # The whole per-GPU host allowance (TOTAL_CPU_DRAM_GB 241). + hicache-size: 241 + max-mamba-cache-size: 448 + max-running-requests: 72 + cuda-graph-max-bs-decode: 128 + # Every size to 80, so decode batches at this concurrency never pad. + cuda-graph-bs-decode: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128] + # Cache only the prompt on finish; the AgentX replay never re-sends model output. + strip-thinking-cache: true + # At most two Mamba states per conversation path; stale interior states skip host write-back. + mamba-max-states-per-path: 2 + # The concurrency-56 admission merging, triggered at 10% of running, with eight decode rounds between prefills. + enable-prefill-delayer: true + prefill-delayer-queue-min-ratio: 0.1 + prefill-delayer-max-delay-ms: 300 + prefill-max-requests: 4 + prefill-decode-interval: 8 + prefill-decode-interval-min-bs: 12 + # FP8 QSA indexer: the sparse-attention indexer keeps its keys in FP8 and scores with FP8 GEMMs. + qsa-indexer-dtype: fp8_e4m3 benchmark: env: - CONC: '64' + CONC: '80' diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index e780521744..ec87e28d02 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -6904,9 +6904,8 @@ dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg: ep: 32 dp-attn: true -# One serving GPU, stock ARM64 release, native AgentX client and proportional host cache. qwen3.8next-fp4-gb300-sglang-agentic-mtp: - image: lmsysorg/sglang:v0.5.22-cu130@sha256:06441be3de19a816ddbac917535e920cbfc7fc43171abc08417394f43634ffb6 + image: lmsysorg/sglang:nightly-dev-cu13-20261103-797af717@sha256:c7c348bc52c410f0e367a46a2a733146017e3de9709694cd42f004fe5d6c763f model: nvidia/Qwen3.8-Flash-Next-NVFP4 model-prefix: qwen3.8next runner: cluster:gb300-nv @@ -6916,19 +6915,12 @@ qwen3.8next-fp4-gb300-sglang-agentic-mtp: srt-recipe-dir: qwen3.8next/sglang/gb300-fp4-mtp scenarios: agentic-coding: - # Use the GB300 profile's proportional per-worker host DRAM allowance. - # Generated allowance: 225 GB. Recipe H224 reserves alignment headroom. + # HiCache rows take the GB300 profile's proportional per-GPU host DRAM allowance (241 GB). - dram-utilization: 1.0 search-space: - - tp: 1 - require-power: true - ep: 1 - dp-attn: false - spec-decoding: mtp - kv-offloading: dram - kv-offload-backend: { name: hicache } - conc-list: [1, 2, 4, 8, 16, 32, 48, 64] - srt-recipe: agentic.yaml + - { tp: 4, ep: 4, dp-attn: false, require-power: true, spec-decoding: mtp, kv-offloading: none, conc-list: [3], srt-recipe: agentic.yaml } + - { tp: 1, ep: 1, dp-attn: false, require-power: true, spec-decoding: mtp, kv-offloading: none, conc-list: [8, 20], srt-recipe: agentic.yaml } + - { tp: 1, ep: 1, dp-attn: false, require-power: true, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [36, 56, 80], srt-recipe: agentic.yaml } musespark-fp4-gb300-sglang-agentic-dflash: image: lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 1e8edb5c0b..98417fbec8 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12352,3 +12352,12 @@ description: - "Sweep H3 C1 latency on 1, 2, 4 and 8 MI355X GPUs with SGLang Ulysses across all GPUs; the MI355X recipe now sets the Ulysses degree from the GPU count, which keeps the 4-GPU point unchanged." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4818 + +- config-keys: + - qwen3.8next-fp4-gb300-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Update to lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 and enable SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION and breakable prefill CUDA graphs." + - "Replace the search space with TP4/EP4 at concurrency 3, TP1/EP1 at concurrency 8 and 20, and TP1/EP1 with HiCache at concurrency 36, 56 and 80; concurrency 20, 36, 56 and 80 use fp8_e4m3 kv-cache-dtype and speculative-draft-kv-cache-dtype (was bfloat16), concurrency 20, 56 and 80 set qsa-indexer-dtype fp8_e4m3, and concurrency 36 and 80 use 2 speculative steps instead of 3." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4787 From c35c541ab66b008cfb502eab11a5b94714555738 Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 06:55:01 +0800 Subject: [PATCH 017/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20MI300X=20vLLM=20AgentX=20DSpark=20image=20to=20v?= =?UTF-8?q?0.33.0=20/=20[Klaud=20Cold]=20=E5=B0=86=20DeepSeek-V4.1-Flash?= =?UTF-8?q?=20FP4=20MI300X=20vLLM=20AgentX=20DSpark=20=E9=95=9C=E5=83=8F?= =?UTF-8?q?=E6=9B=B4=E6=96=B0=E8=87=B3=20v0.33.0=20(#4793)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(dsv41flash): update MI300X vLLM AgentX image to v0.33.0 Co-Authored-By: Claude Opus 5.5 * chore(dsv41flash): add MI300X vLLM AgentX v0.33.0 perf-changelog entry Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/amd-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index e8c34a2cda..3d87122d4a 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -670,7 +670,7 @@ dsr1-fp4-mi355x-sglang-disagg-mtp: # golden AL 3.51 for throughput, real block verification for evals, adaptive # verification off on ROCm. dsv41flash-fp4-mi300x-vllm-agentic-dspark: - image: vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905 + image: vllm/vllm-openai-rocm:v0.33.0@sha256:4fcd39b758b660c4e9c345c021e2ea8f0ccdb31e3c32507d676cc3005ae83fba model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi300x-amd diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 98417fbec8..6d6c09090b 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12361,3 +12361,10 @@ - "Update to lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 and enable SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION and breakable prefill CUDA graphs." - "Replace the search space with TP4/EP4 at concurrency 3, TP1/EP1 at concurrency 8 and 20, and TP1/EP1 with HiCache at concurrency 36, 56 and 80; concurrency 20, 36, 56 and 80 use fp8_e4m3 kv-cache-dtype and speculative-draft-kv-cache-dtype (was bfloat16), concurrency 20, 56 and 80 set qsa-indexer-dtype fp8_e4m3, and concurrency 36 and 80 use 2 speculative steps instead of 3." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4787 + +- config-keys: + - dsv41flash-fp4-mi300x-vllm-agentic-dspark + scenario-type: [agentic-coding] + description: + - "Update the DeepSeek-V4.1-Flash FP4 MI300X vLLM AgentX DSpark image from v0.32.0 to v0.33.0." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4793 From 597076393bab115758dd97534afebad9a056f6e5 Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 07:08:52 +0800 Subject: [PATCH 018/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20GB200=20SGLang=20AgentX=20DSpark=20image=20to=20?= =?UTF-8?q?nightly-dev-cu13-20261027-5884e4fe=20/=20[Klaud=20Cold]=20?= =?UTF-8?q?=E5=B0=86=20DeepSeek-V4.1-Flash=20FP4=20GB200=20SGLang=20AgentX?= =?UTF-8?q?=20DSpark=20=E9=95=9C=E5=83=8F=E6=9B=B4=E6=96=B0=E8=87=B3=20nig?= =?UTF-8?q?htly-dev-cu13-20261027-5884e4fe=20(#4591)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(dsv41flash): update GB200 SGLang AgentX DSpark image to nightly-dev-cu13-20261027-5884e4fe Co-Authored-By: Claude Opus 5.5 * perf(dsv41flash): add perf-changelog entry for GB200 SGLang AgentX DSpark image update Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index ec87e28d02..cb6723dca1 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5857,7 +5857,7 @@ dsv41flash-fp4-gb200-vllm-agentic-dspark: # --speculative-dspark-block-size 5), thinking on, golden AL 3.51 for # throughput, real verification for evals. dsv41flash-fp4-gb200-sglang-agentic-dspark: - image: lmsysorg/sglang:nightly-dev-cu13-20261025-65633129@sha256:069efa1d6a2d58309529f53eeee34c39d69b1cdb455eaf7fb8de83e3fd0be727 + image: lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe@sha256:61b9083bdcb7cf3741366f487cb996c05ea33ad2ddddc820d10cbc1c0cfb0a86 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:gb200-nv diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 6d6c09090b..a8dfbf51b5 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12368,3 +12368,10 @@ description: - "Update the DeepSeek-V4.1-Flash FP4 MI300X vLLM AgentX DSpark image from v0.32.0 to v0.33.0." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4793 + +- config-keys: + - dsv41flash-fp4-gb200-sglang-agentic-dspark + scenario-type: [agentic-coding] + description: + - "Update the SGLang image from nightly-dev-cu13-20261025-65633129 to nightly-dev-cu13-20261027-5884e4fe." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4591 From b0cde4846eba88c29a9ba45d28faa5fe2d28fb05 Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 07:15:30 +0800 Subject: [PATCH 019/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20H200=20SGLang=20AgentX=20DSpark=20image=20to=20n?= =?UTF-8?q?ightly-dev-cu13-20261028-586355a8=20/=20[Klaud=20Cold]=20?= =?UTF-8?q?=E5=B0=86=20DeepSeek-V4.1-Flash=20FP4=20H200=20SGLang=20AgentX?= =?UTF-8?q?=20DSpark=20=E9=95=9C=E5=83=8F=E6=9B=B4=E6=96=B0=E8=87=B3=20nig?= =?UTF-8?q?htly-dev-cu13-20261028-586355a8=20(#4597)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(dsv41flash): update H200 SGLang AgentX DSpark image to nightly-dev-cu13-20261028-586355a8 Co-Authored-By: Claude Opus 5.5 * chore(dsv41flash): add perf-changelog entry for H200 SGLang AgentX DSpark image update Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index cb6723dca1..a6e2206a28 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -6248,7 +6248,7 @@ dsv41flash-fp4-h200-vllm-agentic-dspark: # Match published vLLM TP4/EP1 and TP8/EP1 using dsv4 attention and the nightly's # native MXFP4 x BF16 Marlin MoE runner. dsv41flash-fp4-h200-sglang-agentic-dspark: - image: lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1 + image: lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8@sha256:b4f09a2ccbcfcc262d49600c3cd7f8581a793ea7d104947257b6a95500953b90 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:h200-dgxc diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index a8dfbf51b5..063b059f2f 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12375,3 +12375,10 @@ description: - "Update the SGLang image from nightly-dev-cu13-20261025-65633129 to nightly-dev-cu13-20261027-5884e4fe." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4591 + +- config-keys: + - dsv41flash-fp4-h200-sglang-agentic-dspark + scenario-type: [agentic-coding] + description: + - "Update the H200 SGLang image from nightly-dev-cu13-20261013-7224fce1 to nightly-dev-cu13-20261028-586355a8." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4597 From a93c703dec7178dd8b86167e2a270722344baeae Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 07:22:05 +0800 Subject: [PATCH 020/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20DeepSeek-V4.1?= =?UTF-8?q?-Flash=20FP4=20B200=20SGLang=20AgentX=20DSpark=20image=20to=20n?= =?UTF-8?q?ightly-dev-cu13-20261029-c10e0e17=20/=20[Klaud=20Cold]=20?= =?UTF-8?q?=E5=B0=86=20DeepSeek-V4.1-Flash=20FP4=20B200=20SGLang=20AgentX?= =?UTF-8?q?=20DSpark=20=E9=95=9C=E5=83=8F=E6=9B=B4=E6=96=B0=E8=87=B3=20nig?= =?UTF-8?q?htly-dev-cu13-20261029-c10e0e17=20(#4635)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(configs): update DeepSeek-V4.1-Flash FP4 B200 SGLang AgentX image to nightly-dev-cu13-20261029-c10e0e17 Co-Authored-By: Claude Opus 5.5 * chore(perf-changelog): record DeepSeek-V4.1-Flash FP4 B200 SGLang AgentX image update Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index a6e2206a28..602e92fe7e 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -5979,7 +5979,7 @@ dsv41flash-fp4-b200-vllm-agentic-dspark: - { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [224, 256, 384, 512, 544, 576], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml } dsv41flash-fp4-b200-sglang-agentic-dspark: - image: lmsysorg/sglang:nightly-dev-cu13-20261026-8252e3af@sha256:b4e4abb9e1dc038f7e14c85805be97e4f74abb46e541bd4edd31401d50a2e593 + image: lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17@sha256:78142f9f4b7fb3ee27f59065615dc7bb263f643ed256c9299c3861086ade4d47 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:b200-nscale diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 063b059f2f..5bd5219412 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12382,3 +12382,10 @@ description: - "Update the H200 SGLang image from nightly-dev-cu13-20261013-7224fce1 to nightly-dev-cu13-20261028-586355a8." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4597 + +- config-keys: + - dsv41flash-fp4-b200-sglang-agentic-dspark + description: + - "Update the SGLang image from nightly-dev-cu13-20261026-8252e3af to nightly-dev-cu13-20261029-c10e0e17." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4635 + scenario-type: [agentic-coding] From a780b76679693cefcefd33ef604657ef1b012cb6 Mon Sep 17 00:00:00 2001 From: Klaud Cold Date: Sat, 7 Nov 2026 07:28:42 +0800 Subject: [PATCH 021/111] =?UTF-8?q?[Klaud=20Cold]=20Update=20GLM-5.2=20FP4?= =?UTF-8?q?=20B200=20SGLang=20AgentX=20MTP=20image=20to=20nightly-dev-cu13?= =?UTF-8?q?-20261027-5884e4fe=20/=20[Klaud=20Cold]=20=E5=B0=86=20GLM-5.2?= =?UTF-8?q?=20FP4=20B200=20SGLang=20AgentX=20MTP=20=E9=95=9C=E5=83=8F?= =?UTF-8?q?=E6=9B=B4=E6=96=B0=E8=87=B3=20nightly-dev-cu13-20261027-5884e4f?= =?UTF-8?q?e=20(#4573)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(glm5.2): update FP4 B200 SGLang AgentX MTP image to nightly-dev-cu13-20261027-5884e4fe Co-Authored-By: Claude Opus 5.5 * chore(glm5.2): add perf-changelog entry for FP4 B200 SGLang AgentX MTP image update Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> --- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 602e92fe7e..96f5291464 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -4653,7 +4653,7 @@ glm5.2-fp8-b300-sglang-agentic-mtp: # memory as well as comparability -- the ~433 GB NVFP4 # checkpoint needs ~54 GB/GPU across 8 B200s and does not fit below 8. glm5.2-fp4-b200-sglang-agentic-mtp: - image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae + image: lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe@sha256:61b9083bdcb7cf3741366f487cb996c05ea33ad2ddddc820d10cbc1c0cfb0a86 model: nvidia/GLM-5.2-NVFP4 model-prefix: glm5.2 runner: cluster:b200-nscale diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 5bd5219412..31ee165e58 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12389,3 +12389,11 @@ - "Update the SGLang image from nightly-dev-cu13-20261026-8252e3af to nightly-dev-cu13-20261029-c10e0e17." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4635 scenario-type: [agentic-coding] + +- config-keys: + - glm5.2-fp4-b200-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Update SGLang image from v0.5.22-cu130 to nightly-dev-cu13-20261027-5884e4fe." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4573 From 2a5067e648bcf6a94dfdeaaa652e4acc2582c313 Mon Sep 17 00:00:00 2001 From: hshrivastava-droid Date: Fri, 6 Nov 2026 16:11:47 -0800 Subject: [PATCH 022/111] =?UTF-8?q?[NV]=20Qwen3.8-Flash-Next=20B200=20FP4?= =?UTF-8?q?=20SGLang=20AgentX=20update=20/=20[NV]=20Qwen3.8-Flash-Next=20B?= =?UTF-8?q?200=20FP4=20SGLang=20AgentX=20=E6=9B=B4=E6=96=B0=20(#4850)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Qwen3.8-Flash-Next B200 FP4 SGLang AgentX Co-Authored-By: Claude Sonnet 5.5 * Update perf-changelog PR link Co-Authored-By: Claude Sonnet 5.5 * Remove Qwen3.8-Flash-Next B200 FP4 SGLang AgentX concurrency 1 and 4 Co-Authored-By: Claude Sonnet 5.5 --------- Co-authored-by: Claude Sonnet 5.5 Co-authored-by: Ankur Singh Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .../sglang/b200-fp4-mtp/agentic.yaml | 90 ++++++++++++++----- inferencex-e2e/configs/nvidia-master.yaml | 6 +- inferencex-e2e/perf-changelog.yaml | 9 ++ 3 files changed, 83 insertions(+), 22 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/b200-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/b200-fp4-mtp/agentic.yaml index eca1d3b08a..5fbd094036 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/b200-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/b200-fp4-mtp/agentic.yaml @@ -44,6 +44,7 @@ base: NCCL_NVLS_ENABLE: '1' SGL_ENABLE_JIT_DEEPGEMM: 'false' SGLANG_ENABLE_FLASHINFER_GEMM: 'true' + SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '0' # Outlast AIPerf's 300 s client connection pool. SGLANG_TIMEOUT_KEEP_ALIVE: '1800' benchmark: @@ -51,49 +52,98 @@ base: AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' # One variant per point. NEXTN resets an unset admission cap to 48, so it is -# explicit: 2x CONC, with the graph batch equal to it. C16 needs a larger -# static fraction and a bounded Mamba cache. -override_c1: +# explicit at every point. +override_c8: roles: agg: args: - max-running-requests: 2 - cuda-graph-max-bs: 2 + max-running-requests: 16 + cuda-graph-max-bs-decode: 32 mem-fraction-static: 0.8 benchmark: env: - CONC: '1' + CONC: '8' -override_c4: +override_tp2_c8: roles: agg: + gpus: 2 args: - max-running-requests: 8 - cuda-graph-max-bs: 8 + tensor-parallel-size: 2 + max-running-requests: 16 + cuda-graph-max-bs-decode: 16 mem-fraction-static: 0.8 benchmark: env: - CONC: '4' + CONC: '8' -override_c8: +override_dram_c16: roles: agg: args: - max-running-requests: 16 - cuda-graph-max-bs: 16 - mem-fraction-static: 0.8 + max-running-requests: 32 + cuda-graph-max-bs-decode: 32 + cuda-graph-backend-prefill: disabled + mem-fraction-static: 0.85 + kv-cache-dtype: fp8_e4m3 + mamba-radix-cache-strategy: extra_buffer_lazy + max-mamba-cache-size: 320 + speculative-num-steps: 2 + speculative-num-draft-tokens: 3 + enable-hierarchical-cache: true + hicache-size: 200 + hicache-io-backend: kernel + hicache-mem-layout: page_first + hicache-write-policy: write_back benchmark: env: - CONC: '8' + CONC: '16' + KV_OFFLOADING: dram + TOTAL_CPU_DRAM_GB: '214' -override_c16: +override_dram_c24: roles: agg: args: max-running-requests: 32 - cuda-graph-max-bs: 32 - mem-fraction-static: 0.9 - max-mamba-cache-size: 160 + cuda-graph-max-bs-decode: 32 + cuda-graph-backend-prefill: disabled + mem-fraction-static: 0.85 + kv-cache-dtype: fp8_e4m3 + mamba-radix-cache-strategy: extra_buffer_lazy + max-mamba-cache-size: 320 + speculative-num-steps: 2 + speculative-num-draft-tokens: 3 + prefill-decode-interval: 16 + enable-hierarchical-cache: true + hicache-size: 200 + hicache-io-backend: kernel + hicache-mem-layout: page_first + hicache-write-policy: write_back benchmark: env: - CONC: '16' + CONC: '24' + KV_OFFLOADING: dram + TOTAL_CPU_DRAM_GB: '214' + +override_dram_c32: + roles: + agg: + args: + max-running-requests: 32 + cuda-graph-max-bs-decode: 32 + cuda-graph-backend-prefill: disabled + mem-fraction-static: 0.85 + kv-cache-dtype: fp8_e4m3 + mamba-radix-cache-strategy: extra_buffer_lazy + max-mamba-cache-size: 320 + enable-hierarchical-cache: true + hicache-size: 200 + hicache-io-backend: kernel + hicache-mem-layout: page_first + hicache-write-policy: write_back + benchmark: + env: + CONC: '32' + KV_OFFLOADING: dram + TOTAL_CPU_DRAM_GB: '214' diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 96f5291464..857008156d 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -4154,7 +4154,7 @@ qwen3.5-fp4-b200-sglang-agentic-mtp: # Qwen3.8-Flash-Next NVFP4 AgentX on B200 via SGLang with native NEXTN MTP. # TP1 matches the verified single-node recipe. qwen3.8next-fp4-b200-sglang-agentic-mtp: - image: lmsysorg/sglang:qwen38flashnext + image: lmsysorg/sglang:v0.5.23-cu130@sha256:f832b5c39559c9caa900b22ad37c76fcce66470676855c42052102c537812f1f model: RadixArk/Qwen3.8-Flash-Next-NVFP4 model-prefix: qwen3.8next runner: cluster:b200-nscale @@ -4166,7 +4166,9 @@ qwen3.8next-fp4-b200-sglang-agentic-mtp: agentic-coding: - dram-utilization: 0.8 search-space: - - { tp: 1, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16], srt-recipe: agentic.yaml } + - { tp: 1, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [8], srt-recipe: agentic.yaml } + - { tp: 2, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [8], srt-recipe: agentic.yaml } + - { tp: 1, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [16, 24, 32], srt-recipe: agentic.yaml } qwen3.5-fp4-gb200-dynamo-sglang-agentic-mtp: image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae model: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 31ee165e58..5a6363e66b 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12397,3 +12397,12 @@ description: - "Update SGLang image from v0.5.22-cu130 to nightly-dev-cu13-20261027-5884e4fe." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4573 + +- config-keys: + - qwen3.8next-fp4-b200-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Update to lmsysorg/sglang:v0.5.23-cu130 and set SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0." + - "Remove TP1/EP1 at concurrency 1 and 4 and replace the search space with TP1/EP1 at concurrency 8, TP2/EP1 at concurrency 8, and TP1/EP1 with HiCache at concurrency 16, 24 and 32; the HiCache points use fp8_e4m3 kv-cache-dtype, and concurrency 16 and 24 use 2 speculative steps instead of 3." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4850 From db2af24a9df51465f2a54d01f3460970f98d2622 Mon Sep 17 00:00:00 2001 From: zijie Date: Fri, 6 Nov 2026 22:40:40 -0700 Subject: [PATCH 023/111] [AMD][MI35X] 1105 Kimi K3.1 Pro sglang dflash agentic (#4846) * [AMD][MI35X] 1105 Kimi K3.1 Pro sglang dflash agentic - Image: lmsysorg/sglang-rocm:dev-kimi-k31-pro-mi35x-1105 (day-0 AMD SGLang with #48437, #48439 and #48440; sgl-amd-mono e52803f9d2). - Concurrency 1 and 4: max-mamba-cache-size 60 -> 150. * Link perf-changelog entry to #4846 * Row-sharded MoE front and stream at concurrency 16, 32 and 48 * Keep only the reason in the Mamba cache comment * Drop the row-sharded MoE front and stream at concurrency 16, 32 and 48 * Concurrency 48: 32k prefill chunks every 8 decode steps, row-sharded residual stream Co-Authored-By: Claude Opus 5.5 * Concurrency 48: back to 16k prefill chunks every 16 decode steps Both c48 attempts with 32k chunks every 8 decode steps were killed mid-profile (run 42161676877, nodes mia1-p01-g16 and mia1-p01-g09) with no traceback. Keep the row-sharded residual stream, which costs no memory. Co-Authored-By: Claude Opus 5.5 * New image; concurrency 48 and 56: 5-token verify, 32k chunks every 8 decode steps lmsysorg/sglang:dev-kimi-k31-pro-mi35x (sha256:346a6d83) carries SGLang #48427 at 64507491d9: DCP shares the TP communicators (about 9 GiB a GPU back), the latent-MoE tail column-splits up to 384 rows, the radix-4 router, and ROCm host pools sized against the amdgpu pinned-memory limit; sgl-amd-mono 25723f1e46. Concurrency 48, and a new 56: DFLASH verifies the current token plus 4 drafts (golden AL 3.25), 32k prefill chunks every 8 decode steps, the MoE front and residual stream on row shards in prefill, and fused draft ring locations. The c48 kills of run 42161676877 were on the 1105 image; on this code both points ran 3600 s on 1x8 MI355X (simulated acceptance): c48 19,233 tok/s/GPU at memory fraction 0.826, c56 19,507 at 0.85. Co-Authored-By: Claude Opus 5.5 * Concurrency 8 to 56: Mamba host ratio 12 -> 10 The new image fits about 20% more KV and Kimi state slots on each GPU, and the HiCache host pools follow them: about 196 GB pinned per GPU at concurrency 56 and more at 32, against 1511 GiB of GTT the node's 8 ranks share. Concurrency 32 failed its hipHostMalloc of a 144.61 GB state pool (run 42322137350, mia1-p01-g09). Ten host states per device state leave 170-185 GB a GPU. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com> Co-authored-by: Kevin Mi Co-authored-by: Claude Opus 5.5 --- .../sglang/mi355x-fp4-mtp/agentic.yaml | 57 ++++++++++++++++--- inferencex-e2e/configs/amd-master.yaml | 4 +- inferencex-e2e/perf-changelog.yaml | 13 +++++ 3 files changed, 65 insertions(+), 9 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/sglang/mi355x-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/sglang/mi355x-fp4-mtp/agentic.yaml index 26cc889b37..42286764a2 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/sglang/mi355x-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/sglang/mi355x-fp4-mtp/agentic.yaml @@ -63,6 +63,8 @@ base: ROCM_QUICK_REDUCE_QUANTIZATION: NONE PYTORCH_HIP_ALLOC_CONF: garbage_collection_threshold:0.95 SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK: '1' + # Split-K HIP sparse MLA decode and verify for 16 to 512 rows. + SGLANG_ROCM_SPARSE_MLA_DECODE_HIP: '1' # Pinned host pools; registered pageable pools stall every GPU queue on page migration. HSA_USERPTR_FOR_PAGED_MEM: '0' # The MI355X nodes report MEC firmware 38, below the 177 that fixes the @@ -81,7 +83,8 @@ override_tp8_c1: enable-amd-monokernel: true max-running-requests: 4 cuda-graph-max-bs-decode: 4 - max-mamba-cache-size: 60 + # 150 state slots keep agent sessions' prefixes cached. + max-mamba-cache-size: 150 env: SGLANG_AMD_MONO_K31_WIDE: '1' SGLANG_ROCM_RUNTIME_RESERVE_GB: '6' @@ -97,7 +100,8 @@ override_tp8_c4: enable-amd-monokernel: true max-running-requests: 4 cuda-graph-max-bs-decode: 4 - max-mamba-cache-size: 60 + # 150 state slots keep agent sessions' prefixes cached. + max-mamba-cache-size: 150 env: SGLANG_AMD_MONO_K31_WIDE: '1' SGLANG_ROCM_RUNTIME_RESERVE_GB: '6' @@ -120,8 +124,8 @@ override_tp8_dcp8_c8: cuda-graph-max-bs-decode: 8 env: SGLANG_ROCM_RUNTIME_RESERVE_GB: '10' - # About 190 GB of pinned host pools per GPU, within the node's 1511 GiB GTT limit. - SGLANG_HICACHE_MAMBA_HOST_RATIO: '12' + # 170-185 GB of pinned host pools per GPU: the 8 ranks share the node's 1511 GiB GTT. + SGLANG_HICACHE_MAMBA_HOST_RATIO: '10' # Keeps every rank on the Rust TreeCore; write-back asserts on a mix with the Python one. RUSTUP_TOOLCHAIN: stable benchmark: @@ -143,7 +147,7 @@ override_tp8_dcp8_c16: cuda-graph-max-bs-decode: 16 env: SGLANG_ROCM_RUNTIME_RESERVE_GB: '10' - SGLANG_HICACHE_MAMBA_HOST_RATIO: '12' + SGLANG_HICACHE_MAMBA_HOST_RATIO: '10' RUSTUP_TOOLCHAIN: stable benchmark: env: @@ -164,7 +168,7 @@ override_tp8_dcp8_c32: cuda-graph-max-bs-decode: 32 env: SGLANG_ROCM_RUNTIME_RESERVE_GB: '10' - SGLANG_HICACHE_MAMBA_HOST_RATIO: '12' + SGLANG_HICACHE_MAMBA_HOST_RATIO: '10' RUSTUP_TOOLCHAIN: stable benchmark: env: @@ -183,11 +187,50 @@ override_tp8_dcp8_c48: hicache-ratio: 2.5 max-running-requests: 48 cuda-graph-max-bs-decode: 48 + chunked-prefill-size: 32768 + prefill-decode-interval: 8 + # The current token plus 4 drafts: from 48 requests the shorter verify outweighs its lower acceptance. + speculative-num-draft-tokens: 5 env: SGLANG_ROCM_RUNTIME_RESERVE_GB: '10' - SGLANG_HICACHE_MAMBA_HOST_RATIO: '12' + SGLANG_HICACHE_MAMBA_HOST_RATIO: '10' RUSTUP_TOOLCHAIN: stable + # MoE front and residual stream on row shards in prefill (no extra memory). + SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_ROUTED_FRONT: '1' + SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_STREAM: '1' + # A draft block's ring locations in one launch. + SGLANG_OPT_DFLASH_RING_LOCS_FUSED: '1' benchmark: env: CONC: '48' KV_OFFLOADING: dram + +override_tp8_dcp8_c56: + roles: + agg: + args: + dcp-size: 8 + dcp-comm-backend: a2a + mamba-full-memory-ratio: 0.65 + enable-hierarchical-cache: true + hicache-write-policy: write_back + hicache-ratio: 2.5 + max-running-requests: 56 + cuda-graph-max-bs-decode: 56 + chunked-prefill-size: 32768 + prefill-decode-interval: 8 + # The current token plus 4 drafts: from 48 requests the shorter verify outweighs its lower acceptance. + speculative-num-draft-tokens: 5 + env: + SGLANG_ROCM_RUNTIME_RESERVE_GB: '10' + SGLANG_HICACHE_MAMBA_HOST_RATIO: '10' + RUSTUP_TOOLCHAIN: stable + # MoE front and residual stream on row shards in prefill (no extra memory). + SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_ROUTED_FRONT: '1' + SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_STREAM: '1' + # A draft block's ring locations in one launch. + SGLANG_OPT_DFLASH_RING_LOCS_FUSED: '1' + benchmark: + env: + CONC: '56' + KV_OFFLOADING: dram diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 3d87122d4a..02c05727c3 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -1077,7 +1077,7 @@ musespark-fp4-mi355x-sglang-agentic-dflash: - { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [10, 16, 24, 28, 36, 40], srt-recipe: agentic.yaml } kimik3.1-fp4-mi355x-sglang-agentic-dflash: - image: lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:6af91676b98cb98246aab7da6ab5a8cdaa0f30d9e0f4a9d44e6b19c9f99e87a0 + image: lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:346a6d832bbfcb5776d8d938154870682a891278a8adc264ed996ccf3d3b6762 model: moonshotai/Kimi-K3.1-Pro model-prefix: kimik3.1 runner: cluster:mi355x-amds @@ -1090,7 +1090,7 @@ kimik3.1-fp4-mi355x-sglang-agentic-dflash: - dram-utilization: 0.513 search-space: - { tp: 8, ep: 1, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 4], srt-recipe: agentic.yaml } - - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [8, 16, 32, 48], srt-recipe: agentic.yaml } + - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [8, 16, 32, 48, 56], srt-recipe: agentic.yaml } glm5.3flash-fp8-mi325x-sglang-agentic-mtp: image: lmsysorg/sglang:v0.5.22-rocm10-mi30x@sha256:5658cd0d8de0c68de10d2e349296f55ceff98053182fa87b2a7e942e5e7f3c2a diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 5a6363e66b..3473c978b2 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12406,3 +12406,16 @@ - "Update to lmsysorg/sglang:v0.5.23-cu130 and set SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0." - "Remove TP1/EP1 at concurrency 1 and 4 and replace the search space with TP1/EP1 at concurrency 8, TP2/EP1 at concurrency 8, and TP1/EP1 with HiCache at concurrency 16, 24 and 32; the HiCache points use fp8_e4m3 kv-cache-dtype, and concurrency 16 and 24 use 2 speculative steps instead of 3." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4850 + +- config-keys: + - kimik3.1-fp4-mi355x-sglang-agentic-dflash + scenario-type: + - agentic-coding + description: + - "Bump to lmsysorg/sglang-rocm:dev-kimi-k31-pro-mi35x-1105: SGLang kimi-k3.1-day-0-amd with split-K HIP sparse MLA decode/verify, DFLASH fc row-sharding in prefill and MoE small sort up to 1024 pairs; sgl-amd-mono e52803f9d2." + - "Concurrency 1 and 4: max-mamba-cache-size 60 -> 150." + - "Bump to lmsysorg/sglang:dev-kimi-k31-pro-mi35x (sha256:346a6d83): SGLang #48427 at 64507491d9 with DCP sharing the TP communicators, the latent-MoE tail column-split up to 384 rows, the radix-4 router and ROCm host pools sized against the pinned-memory limit; sgl-amd-mono 25723f1e46." + - "Concurrency 48 and 56: 32k prefill chunks every 8 decode steps, 5-token DFLASH verify, MoE front and residual stream on row shards in prefill (SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_ROUTED_FRONT, SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_STREAM), fused DFLASH ring locations." + - "Add concurrency 56." + - "Concurrency 8 to 56: SGLANG_HICACHE_MAMBA_HOST_RATIO 12 -> 10; the image's larger device pools grew the pinned host pools past the node's GTT (hipHostMalloc failed at concurrency 32)." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4846 From 2b7b1d96fa9a3cce2fd2cda7c8ff43e60e1ef5b3 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 00:47:04 -0500 Subject: [PATCH 024/111] Mark README as experimental --- video-generation/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/video-generation/README.md b/video-generation/README.md index e96dd17a55..cb40d4250a 100644 --- a/video-generation/README.md +++ b/video-generation/README.md @@ -1,4 +1,4 @@ -# Video-generation harness +# Video-generation harness [experimental] **English** | [中文](README_zh.md) From 2fd1f009a22bfe8632592f0986364a53848eab24 Mon Sep 17 00:00:00 2001 From: Bryan <58582368+Oseltamivir@users.noreply.github.com> Date: Sat, 7 Nov 2026 15:18:14 +0800 Subject: [PATCH 025/111] =?UTF-8?q?CollectiveX:=20charge=20vLLM's=20finali?= =?UTF-8?q?ze=20copy=20in=20DeepEP=20V2=20and=20UCCL-EP=20combine=20/=20Co?= =?UTF-8?q?llectiveX=EF=BC=9A=E5=9C=A8=20DeepEP=20V2=20=E4=B8=8E=20UCCL-EP?= =?UTF-8?q?=20=E7=9A=84=20combine=20=E4=B8=AD=E8=AE=A1=E5=85=A5=20vLLM=20?= =?UTF-8?q?=E7=9A=84=20finalize=20=E6=8B=B7=E8=B4=9D=20(#4882)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(collectivex): charge vLLM's finalize copy in deepep-v2 and uccl-ep normal combine vLLM v0.32.0's deepep_v2 and deepep_ht finalize copy the combined rows into the MoE layer's output; mori and flashinfer-ep already time that copy, these two returned the buffer. Rows record library_config.finalize_copy. Docs: FlashInfer coverage on H100/H200, DeepEP V2's venv torch/NCCL. vLLM v0.32.0 的 deepep_v2 与 deepep_ht finalize 会把合并结果拷贝到 MoE 层输出；mori 和 flashinfer-ep 已计入该拷贝，这两个后端此前直接返回缓冲区。结果在 library_config.finalize_copy 中记录。文档：补充 FlashInfer 在 H100/H200 的覆盖以及 DeepEP V2 venv 的 torch/NCCL。 * docs(collectivex): record the measured cost of the finalize copy 文档：记录 finalize 拷贝的实测开销。 * docs(collectivex): add the perf-changelog entries for the finalize copy 为 finalize 拷贝添加 perf-changelog 条目。 --- collectivex/bench/ep_deepep_v2.py | 1 + collectivex/bench/ep_legacy_ll.py | 15 +++++++++++++- collectivex/bench/ep_uccl.py | 1 + collectivex/docs/methodology.md | 21 ++++++++++++++------ collectivex/perf-changelog.yaml | 14 +++++++++++++ collectivex/runtime/common.sh | 4 ++-- collectivex/tests/test_deepep_v2_contract.py | 17 ++++++++++++++-- collectivex/tests/test_uccl_contract.py | 11 +++++++--- 8 files changed, 70 insertions(+), 14 deletions(-) diff --git a/collectivex/bench/ep_deepep_v2.py b/collectivex/bench/ep_deepep_v2.py index 2ef3be2f9a..886a190690 100644 --- a/collectivex/bench/ep_deepep_v2.py +++ b/collectivex/bench/ep_deepep_v2.py @@ -298,6 +298,7 @@ def implementation_details(self): "receive_capacity": "next-pow2-of-tokens", "buffer_tokens_per_rank": getattr(self, "max_tokens", None), "zero_copy": False, + "finalize_copy": True, "deepep_source": _DEEPEP_REPO.removeprefix("https://github.com/"), # The fork's scale-out kernel pair (unordered by default); upstream has one. "hybrid_kernel": os.environ.get("EP_HYBRID_KERNEL", "unordered") if _EFA_FORK else "ordered", diff --git a/collectivex/bench/ep_legacy_ll.py b/collectivex/bench/ep_legacy_ll.py index 34e044cf94..452294d66e 100644 --- a/collectivex/bench/ep_legacy_ll.py +++ b/collectivex/bench/ep_legacy_ll.py @@ -159,6 +159,15 @@ def implementation_details(self): }) return details + def make_problem(self, T, idx, weights, x): + """Adds the MoE layer's output for normal mode, which vLLM's deepep_v2 and deepep_ht + finalize copy the combined rows into; allocated once per rung, untimed.""" + problem = super().make_problem(T, idx, weights, x) + if self.mode != "low-latency": + problem.combine_output = torch.empty( + (T, self.args.hidden), dtype=torch.bfloat16, device=self.device) + return problem + # ---- transport contract, routed by mode ---------------------------------------------- def create_buffer(self, spec): @@ -184,7 +193,11 @@ def combine(self, p, h): if self.mode == "low-latency": # The kernel applies topk_weights internally (weighted). return self._ll_combine(p, h, h.combine_input) - return self._normal_combine(p, h) + # vLLM's finalize copies the result into the MoE layer's output (deepep_v2.py, + # deepep_ht.py), so the copy is charged here as mori and flashinfer-ep charge theirs. + # Low-latency combines straight into the output (`out=`), so it has no copy. + p.combine_output.copy_(self._normal_combine(p, h)[:p.T]) + return p.combine_output def inspect_dispatch(self, p, h): if self.mode == "low-latency": diff --git a/collectivex/bench/ep_uccl.py b/collectivex/bench/ep_uccl.py index b84125b9ed..2052a68f5f 100644 --- a/collectivex/bench/ep_uccl.py +++ b/collectivex/bench/ep_uccl.py @@ -279,6 +279,7 @@ def _record_buffer(self, **arguments): # import checked. The proxies exist either way but start only across nodes. proxies = len(getattr(self.buffer, "proxies", ())) if self._internode else 0 self._library_config = {"deep_ep": "uccl deep_ep_wrapper", **arguments, + "finalize_copy": self.mode != "low-latency", "is_intranode": not self._internode, "cpu_proxy_threads": proxies} def implementation_details(self) -> dict: diff --git a/collectivex/docs/methodology.md b/collectivex/docs/methodology.md index 0bee42b6c6..a785507479 100644 --- a/collectivex/docs/methodology.md +++ b/collectivex/docs/methodology.md @@ -32,7 +32,7 @@ library only. Details are under [Backends](#backends). | nccl-ep | production for in-domain zero-copy LL rank-major BF16; library otherwise | graph decode; prefill eager | TensorRT-LLM `NcclEP` with NVIDIA/TensorRT-LLM#20631; HT and FP8 (DS_FP8E3M4) are library rows | | uccl-ep | candidate | normal eager; LL graph | vLLM's DeepEP configuration on UCCL's drop-in `deep_ep`; no engine ships its selector | | mori | production | graph decode; eager prefill | vLLM's MoRI IntraNode / InterNodeV1, LL EP16 only | -| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration, GB200/GB300 only | +| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration; EP8 on H100/H200, EP8 and EP16 on GB200/GB300 | ## Timing @@ -191,7 +191,10 @@ graphs on, vLLM's default, every batch, prefill as well as decode, dispatches wi and `do_expand=False` into a receive bounded at the next power of two of T, with `prefer_overlap_with_compute=False` (DeepEP then sizes its kernels at 64 SMs) and `allow_multiple_reduction=False`. No deepep-v2 dispatch waits on the host. Decode replays a graph and -prefill runs eager, as vLLM runs large prefill batches uncaptured. +prefill runs eager, as vLLM runs large prefill batches uncaptured. `combine` copies the batch's rows +into the MoE layer's output, as vLLM's finalize does (`library_config.finalize_copy`); rows without +that field predate the copy: on b200 EP8 it adds 4-13% to decode `combine`, 4-5% to prefill `combine` +and 1-2% to the prefill pair period (1-5% for uccl-ep). Scale-up cases request NCCL Device API LSA and fail closed unless the LSA team covers the whole EP world, which is also what makes the hybrid choice match vLLM's autoselect: DeepEP goes hybrid only @@ -233,7 +236,8 @@ under vLLM's DeepEP backends, and the adapter runs that configuration. `normal` is vLLM's `deepep_ht` path: UCCL's default 20 SMs, `get_dispatch_config`/`get_combine_config` per call, 1 GiB of NVLink buffer, plus 1 GiB of RDMA buffer and 10 QPs per rank across nodes. Its dispatch polls receive counters on the host before launching the transfer, so `normal` stays eager and -passes through the host gate ([Rank Alignment](#rank-alignment)). `low-latency` issues the same calls +passes through the host gate ([Rank Alignment](#rank-alignment)). Its `combine` carries vLLM's copy +into the layer output, as DeepEP V2's does. `low-latency` issues the same calls as DeepEP V2 (send kernel plus receive hook) with vLLM's 256-token receive; at EP8 `is_intranode` keeps it on NVLink. UCCL-EP is built from source at its pin with UCCL's drop-in flags `USE_DMABUF=1 PER_EXPERT_BATCHING=1`; `implementation.library_version` names the commit and flags, @@ -299,8 +303,9 @@ point, because two handles from one group share its double-buffer parity slots ### FlashInfer EP FlashInfer EP is TensorRT-LLM's one-sided MNNVL `MoeAlltoAll` from FlashInfer 0.7.1, vLLM main's pin: -each rank writes tokens directly into its peers' workspace windows, so it is GB200/GB300-only and runs -EP8 and EP16 inside MNNVL. It has one kernel family, so it runs `normal` only, across both ladders. +each rank writes tokens directly into its peers' workspace windows, so every rank must sit in one +NVLink domain: EP8 on H100 and H200 (one node), EP8 and EP16 inside MNNVL on GB200/GB300. B200 and +B300 are not registered yet. The `flashinfer-mnnvl-…` generation name is kept on the single-node rows. It has one kernel family, so it runs `normal` only, across both ladders. It runs vLLM's integration (`flashinfer_nvlink_one_sided`): combine is handed a caller-owned payload with `payload_in_workspace=False`, so FlashInfer's staging copy into the workspace runs, and is timed, inside `combine`, as in vLLM (writing the expert output into the workspace directly is TensorRT-LLM's @@ -674,7 +679,11 @@ aws-ofi-nccl 1.21.1 where needed. Each cache directory's name binds the CPU arch the pin (plus the GPU architecture and build-recipe generation for GPU builds), and it is never committed. The DeepEP V2 build applies one local patch, the opt-in relaxed-ordering switch; UCCL-EP's AMD build swaps managed allocations for pinned host memory. MoRI ships in the image and is only -import-checked. Runtime probes fail closed before reuse, and a cache that fails them is rejected, not +import-checked. DeepEP V2 does not run on the image's torch: its cache holds an isolated venv with +torch 2.11.0 and NCCL 2.30.4 (2.32.3 for the EFA fork), where the vLLM v0.32.0 image carries torch +2.13.0 and NCCL 2.30.7, which every other NVIDIA backend uses. `runtime.framework` and +`runtime.collective_library` record the pair per row. The pin is 6 commits ahead of the DeepEP commit +vLLM v0.32.0 installs (d4f41e4), which vLLM builds into the image's own environment. Runtime probes fail closed before reuse, and a cache that fails them is rejected, not reset, so a concurrent allocation cannot lose files it is using. ## Result Delivery diff --git a/collectivex/perf-changelog.yaml b/collectivex/perf-changelog.yaml index d78c97875f..c5b4481f66 100644 --- a/collectivex/perf-changelog.yaml +++ b/collectivex/perf-changelog.yaml @@ -137,3 +137,17 @@ suites: [swap-blocks] description: - Take 100 iterations per swap-blocks point, so p99 is no longer the slowest of 20 samples. +- date: 2026-11-07 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4882 + pools: [h100-dgxc, h200-dgxc, b200-nscale, b300, gb200, gb300] + backends: [deepep-v2] + modes: [normal] + description: + - Charge vLLM's finalize copy in DeepEP V2 normal combine (the result is copied into the MoE layer's output, as mori and flashinfer-ep already did). +- date: 2026-11-07 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4882 + pools: [h100-dgxc, h200-dgxc, b200-nscale, mi300x, mi325x, mi355x] + backends: [uccl-ep] + modes: [normal] + description: + - Charge vLLM's finalize copy in UCCL-EP normal combine. diff --git a/collectivex/runtime/common.sh b/collectivex/runtime/common.sh index 2c51fee2aa..2a4d92780d 100644 --- a/collectivex/runtime/common.sh +++ b/collectivex/runtime/common.sh @@ -44,8 +44,8 @@ collx_select_deepep_v2_source() { COLLX_DEEPEP_V2_NVSHMEM_SPEC="nvidia-nvshmem-cu13==3.4.5" # 2.10.0+cu130's bundled CUDA userland poisons the CUDA context during nvshmem symmetric-heap -# init over MNNVL on sm103/driver 580.159.03 (gb300); 2.11.0 is what the cu130 image itself -# ships. Part of the venv cache key. +# init over MNNVL on sm103/driver 580.159.03 (gb300). The vLLM v0.32.0 image ships 2.13.0, which +# this venv does not use. Part of the venv cache key. COLLX_DEEPEP_V2_TORCH_SPEC="torch==2.11.0" # Build-recipe generation for the DeepEP venv cache key: bump when build flags change without diff --git a/collectivex/tests/test_deepep_v2_contract.py b/collectivex/tests/test_deepep_v2_contract.py index 7c9fce2c46..1ecec584fa 100644 --- a/collectivex/tests/test_deepep_v2_contract.py +++ b/collectivex/tests/test_deepep_v2_contract.py @@ -118,8 +118,17 @@ def _built(module, world_size=8, env=None, max_tokens=512, **updates): return backend +class _Output: + """The MoE layer's output: records what combine copied into it.""" + + def copy_(self, value): + self.value = value + return self + + def _problem(tokens): - return types.SimpleNamespace(T=tokens, dispatch_x="x", topk_idx="i", topk_weights="w") + return types.SimpleNamespace(T=tokens, dispatch_x="x", topk_idx="i", topk_weights="w", + combine_output=_Output()) class ElasticBufferContract(unittest.TestCase): @@ -158,7 +167,11 @@ def test_sm_and_qp_counts_are_left_to_deepep_as_vllm_leaves_them(self): # one the kernels run with, and dispatch and combine pass nothing of their own. self.assertEqual(backend.buffer.sm_calls, [((384, 6), {})]) backend.dispatch(_problem(3)) - backend.combine(_problem(3), types.SimpleNamespace(combine_input="staged", handle="h")) + problem = _problem(3) + output = backend.combine(problem, types.SimpleNamespace(combine_input="staged", handle="h")) + # vLLM's finalize copies the batch's rows of the combined plane into the layer output. + self.assertIs(output, problem.combine_output) + self.assertEqual(output.value, "com") dispatched, combined = backend.buffer.dispatches[-1], backend.buffer.combines[-1] self.assertEqual(dispatched["num_sms"], 0) self.assertNotIn("num_qps", dispatched) diff --git a/collectivex/tests/test_uccl_contract.py b/collectivex/tests/test_uccl_contract.py index 9f4a884a82..dc40c65e69 100644 --- a/collectivex/tests/test_uccl_contract.py +++ b/collectivex/tests/test_uccl_contract.py @@ -220,10 +220,15 @@ def test_one_sm_budget_at_ep8_and_ep16_on_cuda_and_hip(self): def test_dispatch_and_combine_pass_the_resolved_configs(self): backend, buffer = _built(self.module) - handle = backend.dispatch(types.SimpleNamespace( - T=4, dispatch_x="x", topk_idx="idx", topk_weights="weights")) + copied = [] + problem = types.SimpleNamespace( + T=4, dispatch_x="x", topk_idx="idx", topk_weights="weights", + combine_output=types.SimpleNamespace(copy_=copied.append)) + handle = backend.dispatch(problem) backend.stage(None, handle) - backend.combine(None, handle) + # vLLM's deepep_ht finalize copies the batch's rows into the layer output. + self.assertIs(backend.combine(problem, handle), problem.combine_output) + self.assertEqual(copied, ["comb"]) (_, dispatched), (_, combined) = buffer.calls self.assertIs(dispatched["config"], backend.dispatch_config) self.assertEqual((dispatched["x"], dispatched["async_finish"]), ("x", False)) From 05a786aff7d2cad1ba63cbd8e527f3385e466651 Mon Sep 17 00:00:00 2001 From: Bryan <58582368+Oseltamivir@users.noreply.github.com> Date: Sat, 7 Nov 2026 16:09:36 +0800 Subject: [PATCH 026/111] =?UTF-8?q?CollectiveX:=20build=20DeepEP=20V2=20on?= =?UTF-8?q?=20the=20image's=20torch=20and=20NCCL;=20run=20FlashInfer=20EP8?= =?UTF-8?q?=20on=20B200=20/=20CollectiveX=EF=BC=9ADeepEP=20V2=20=E6=94=B9?= =?UTF-8?q?=E7=94=A8=E9=95=9C=E5=83=8F=E8=87=AA=E5=B8=A6=E7=9A=84=20torch?= =?UTF-8?q?=20=E4=B8=8E=20NCCL=20=E6=9E=84=E5=BB=BA=EF=BC=9B=E5=9C=A8=20B2?= =?UTF-8?q?00=20=E4=B8=8A=E8=BF=90=E8=A1=8C=20FlashInfer=20EP8=20(#4883)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(collectivex): build DeepEP V2 on the image's torch and NCCL; run flashinfer-ep EP8 on b200 DeepEP V2 的构建改用镜像自带的 torch 与 NCCL；在 b200 上运行 flashinfer-ep EP8。 * docs(collectivex): add the perf-changelog entries 添加 perf-changelog 条目。 --- collectivex/README.md | 8 +- collectivex/configs/platform_config.json | 2 +- collectivex/docs/methodology.md | 18 +- collectivex/perf-changelog.yaml | 12 ++ collectivex/results/cases.json | 228 +++++++++++++++++++++++ collectivex/runtime/common.sh | 18 +- collectivex/runtime/prepare_backend.sh | 32 ++-- 7 files changed, 285 insertions(+), 33 deletions(-) diff --git a/collectivex/README.md b/collectivex/README.md index 57e18b0721..c574511f7e 100644 --- a/collectivex/README.md +++ b/collectivex/README.md @@ -34,15 +34,15 @@ runs the decode and prefill ladders; `low-latency` runs decode, plus prefill for | `mori` | vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` (`production`) | EP8, EP16: MI300X, MI325X, MI355X | EP16: MI300X, MI325X, MI355X | | `uccl-ep` | none; engines load it as a drop-in `deep_ep` under vLLM's DeepEP backends (`candidate`) | EP8: H100, H200, B200, MI300X, MI325X, MI355X. EP16: H100, H200, B200, MI300X, MI325X | EP8: B200. EP8, EP16: H100, H200. EP16: MI300X, MI325X | | `nccl-ep` | TensorRT-LLM `NcclEP` (`production` for low-latency rank-major BF16 where it matches, `candidate` otherwise) | EP8, EP16: H100, H200, B200, B300, GB200, GB300 | EP8: B300. EP8, EP16: H100, H200, B200, GB200, GB300 | -| `flashinfer-ep` | vLLM `--all2all-backend flashinfer_nvlink_one_sided` (`production`) | EP8: H100, H200. EP8, EP16: GB200, GB300 | none | +| `flashinfer-ep` | vLLM `--all2all-backend flashinfer_nvlink_one_sided` (`production`) | EP8: H100, H200, B200. EP8, EP16: GB200, GB300 | none | Every backend runs a BF16 dispatch control and an FP8 dispatch; nccl-ep's FP8 is its internal DS_FP8E3M4 recipe, low-latency prefill and decode. Combine is always BF16. Coverage is uniform routing only. -`results/cases.json` is the committed copy of the full matrix: 284 requested cases, of which 248 are +`results/cases.json` is the committed copy of the full matrix: 290 requested cases, of which 252 are runnable EP cases, 18 are swap-blocks cases (two per pool), 12 are kv-transfer cases (one per pool, -backend and fabric) and 6 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on -H100 and H200, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink +backend and fabric) and 8 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on +H100, H200 and B200, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink domain). A cell is listed as unsupported only when its backend runs on that SKU at another EP size, and low-latency adds only runnable cells, so a missing low-latency cell (DeepEP V2 on B300, for example) diff --git a/collectivex/configs/platform_config.json b/collectivex/configs/platform_config.json index 53888e0c2e..b2164b56c9 100644 --- a/collectivex/configs/platform_config.json +++ b/collectivex/configs/platform_config.json @@ -60,7 +60,7 @@ "scale_up_transport": "nvlink", "launcher": "single-slurm", "runner_label": "cluster:b200-nscale", - "backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8, 16]}, + "backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8, 16], "flashinfer-ep": [8]}, "ll_backends": {"deepep-v2": [8, 16], "uccl-ep": [8], "nccl-ep": [8, 16]}, "kv_backends": {"nixl": ["rdma"], "mooncake": ["rdma"]}, "fabric": {"nic": "8x ConnectX-7-class 400G InfiniBand rails (bare metal, gdrdrv)", "switch": "InfiniBand (model not surveyed)"}, diff --git a/collectivex/docs/methodology.md b/collectivex/docs/methodology.md index a785507479..a343acce6f 100644 --- a/collectivex/docs/methodology.md +++ b/collectivex/docs/methodology.md @@ -32,7 +32,7 @@ library only. Details are under [Backends](#backends). | nccl-ep | production for in-domain zero-copy LL rank-major BF16; library otherwise | graph decode; prefill eager | TensorRT-LLM `NcclEP` with NVIDIA/TensorRT-LLM#20631; HT and FP8 (DS_FP8E3M4) are library rows | | uccl-ep | candidate | normal eager; LL graph | vLLM's DeepEP configuration on UCCL's drop-in `deep_ep`; no engine ships its selector | | mori | production | graph decode; eager prefill | vLLM's MoRI IntraNode / InterNodeV1, LL EP16 only | -| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration; EP8 on H100/H200, EP8 and EP16 on GB200/GB300 | +| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration; EP8 on H100/H200/B200, EP8 and EP16 on GB200/GB300 | ## Timing @@ -304,8 +304,8 @@ point, because two handles from one group share its double-buffer parity slots FlashInfer EP is TensorRT-LLM's one-sided MNNVL `MoeAlltoAll` from FlashInfer 0.7.1, vLLM main's pin: each rank writes tokens directly into its peers' workspace windows, so every rank must sit in one -NVLink domain: EP8 on H100 and H200 (one node), EP8 and EP16 inside MNNVL on GB200/GB300. B200 and -B300 are not registered yet. The `flashinfer-mnnvl-…` generation name is kept on the single-node rows. It has one kernel family, so it runs `normal` only, across both ladders. +NVLink domain: EP8 on H100, H200 and B200 (one node), EP8 and EP16 inside MNNVL on GB200/GB300. B300 +is not registered yet. The `flashinfer-mnnvl-…` generation name is kept on the single-node rows. It has one kernel family, so it runs `normal` only, across both ladders. It runs vLLM's integration (`flashinfer_nvlink_one_sided`): combine is handed a caller-owned payload with `payload_in_workspace=False`, so FlashInfer's staging copy into the workspace runs, and is timed, inside `combine`, as in vLLM (writing the expert output into the workspace directly is TensorRT-LLM's @@ -679,11 +679,13 @@ aws-ofi-nccl 1.21.1 where needed. Each cache directory's name binds the CPU arch the pin (plus the GPU architecture and build-recipe generation for GPU builds), and it is never committed. The DeepEP V2 build applies one local patch, the opt-in relaxed-ordering switch; UCCL-EP's AMD build swaps managed allocations for pinned host memory. MoRI ships in the image and is only -import-checked. DeepEP V2 does not run on the image's torch: its cache holds an isolated venv with -torch 2.11.0 and NCCL 2.30.4 (2.32.3 for the EFA fork), where the vLLM v0.32.0 image carries torch -2.13.0 and NCCL 2.30.7, which every other NVIDIA backend uses. `runtime.framework` and -`runtime.collective_library` record the pair per row. The pin is 6 commits ahead of the DeepEP commit -vLLM v0.32.0 installs (d4f41e4), which vLLM builds into the image's own environment. Runtime probes fail closed before reuse, and a cache that fails them is rejected, not +import-checked. DeepEP V2 builds against the image's torch and NCCL (2.13.0 and 2.30.7 in vLLM +v0.32.0), as vLLM's installer does and as every other NVIDIA backend runs. The EFA fork on b300 is +the exception: it needs NCCL 2.32.3, so its venv carries that wheel and torch 2.11.0. +`runtime.framework` and `runtime.collective_library` record the pair per row; rows reporting torch +2.11.0 with NCCL 2.30.4 predate the change (within 1% on b200 EP8 and 3% on gb300 EP8). The pin is 6 +commits ahead of the DeepEP commit vLLM v0.32.0 packages (d4f41e4); the image's packaged `deep_ep` +matched the pin within 1% on b200 EP8 from T=4 up. Runtime probes fail closed before reuse, and a cache that fails them is rejected, not reset, so a concurrent allocation cannot lose files it is using. ## Result Delivery diff --git a/collectivex/perf-changelog.yaml b/collectivex/perf-changelog.yaml index c5b4481f66..12c0ed1521 100644 --- a/collectivex/perf-changelog.yaml +++ b/collectivex/perf-changelog.yaml @@ -151,3 +151,15 @@ modes: [normal] description: - Charge vLLM's finalize copy in UCCL-EP normal combine. +- date: 2026-11-07 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4883 + pools: [h100-dgxc, h200-dgxc, b200-nscale, gb200, gb300] + backends: [deepep-v2] + description: + - Build DeepEP V2 against the image's torch and NCCL instead of an isolated venv (torch 2.11.0, NCCL 2.30.4). +- date: 2026-11-07 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4883 + pools: [b200-nscale] + backends: [flashinfer-ep] + description: + - Run flashinfer-ep EP8 on B200. diff --git a/collectivex/results/cases.json b/collectivex/results/cases.json index 9eef6bfb65..8527a271c3 100644 --- a/collectivex/results/cases.json +++ b/collectivex/results/cases.json @@ -418,6 +418,82 @@ "reason": null, "sku": "b200-nscale" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-decode-ep8-uniform-bf16", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 1, + "phase": "decode", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b200-nscale" + }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-decode-ep8-uniform-fp8", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 1, + "phase": "decode", + "precision": "fp8", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b200-nscale" + }, { "case": { "backend": "deepep-v2", @@ -684,6 +760,82 @@ "reason": null, "sku": "b200-nscale" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep8-uniform-bf16", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 1, + "phase": "prefill", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b200-nscale" + }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep8-uniform-fp8", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 1, + "phase": "prefill", + "precision": "fp8", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b200-nscale" + }, { "case": { "backend": "deepep-v2", @@ -1026,6 +1178,44 @@ "reason": null, "sku": "b200-nscale" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-decode-ep16-uniform-bf16", + "ep": 16, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 2, + "phase": "decode", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": "rdma", + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-out", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-rdma", + "transport": "nvlink-rdma", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "unsupported", + "reason": "backend-platform-unsupported", + "sku": "b200-nscale" + }, { "case": { "backend": "deepep-v2", @@ -1292,6 +1482,44 @@ "reason": null, "sku": "b200-nscale" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b200-nscale-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep16-uniform-bf16", + "ep": 16, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 2, + "phase": "prefill", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": "rdma", + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-out", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b200-nvlink-rdma", + "transport": "nvlink-rdma", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "unsupported", + "reason": "backend-platform-unsupported", + "sku": "b200-nscale" + }, { "case": { "backend": "deepep-v2", diff --git a/collectivex/runtime/common.sh b/collectivex/runtime/common.sh index 2a4d92780d..3e9dcf7d02 100644 --- a/collectivex/runtime/common.sh +++ b/collectivex/runtime/common.sh @@ -21,10 +21,11 @@ COLLX_DEEPEP_V2_COMMIT="01dc3aaac82068020353dce2c302e38153c0bfaa" # treats EFA as unsupported (deepseek-ai/DeepEP#369). COLLX_DEEPEP_V2_EFA_REPO="https://github.com/amazon-contributing/DeepEP" COLLX_DEEPEP_V2_EFA_COMMIT="a227152ac29513b1c702860745a1d1e0111ae0dc" -# ElasticBuffer needs NCCL 2.30.4 (torch pins 2.28.9). The fork sets GIN signal requirements -# (ginStrongSignalsRequired, ginVaSignalsRequired) that NCCL first declares in 2.32, which is what -# AWS tests it with. Part of the venv cache key. -COLLX_DEEPEP_V2_NCCL_SPEC="nvidia-nccl-cu13==2.30.4" +# ElasticBuffer needs NCCL >= 2.30.4. "image" uses the image's own (vLLM v0.32.0 carries 2.30.7), +# as vLLM's installer does. The fork sets GIN signal requirements (ginStrongSignalsRequired, +# ginVaSignalsRequired) that NCCL first declares in 2.32, so it installs that wheel into its venv. +# Part of the venv cache key. +COLLX_DEEPEP_V2_NCCL_SPEC="image" COLLX_DEEPEP_V2_EFA_NCCL_SPEC="nvidia-nccl-cu13==2.32.3" # Selects this pool's DeepEP V2 source into COLLX_DEEPEP_V2_REPO/COMMIT/NCCL_SPEC (the venv cache is keyed @@ -36,6 +37,7 @@ collx_select_deepep_v2_source() { COLLX_DEEPEP_V2_REPO="$COLLX_DEEPEP_V2_EFA_REPO" COLLX_DEEPEP_V2_COMMIT="$COLLX_DEEPEP_V2_EFA_COMMIT" COLLX_DEEPEP_V2_NCCL_SPEC="$COLLX_DEEPEP_V2_EFA_NCCL_SPEC" + COLLX_DEEPEP_V2_TORCH_SPEC="torch==2.11.0" } # Must match the image's CUDA line: the cu12 wheel's r12 host library on cu130 images survives @@ -43,14 +45,14 @@ collx_select_deepep_v2_source() { # (gb300); every CUDA call after buffer creation fails cudaErrorUnknown. Part of the venv cache key. COLLX_DEEPEP_V2_NVSHMEM_SPEC="nvidia-nvshmem-cu13==3.4.5" +# "image" builds against the image's torch (vLLM v0.32.0: 2.13.0). The fork's venv pins its own: # 2.10.0+cu130's bundled CUDA userland poisons the CUDA context during nvshmem symmetric-heap -# init over MNNVL on sm103/driver 580.159.03 (gb300). The vLLM v0.32.0 image ships 2.13.0, which -# this venv does not use. Part of the venv cache key. -COLLX_DEEPEP_V2_TORCH_SPEC="torch==2.11.0" +# init over MNNVL on sm103/driver 580.159.03 (gb300). Part of the venv cache key. +COLLX_DEEPEP_V2_TORCH_SPEC="image" # Build-recipe generation for the DeepEP venv cache key: bump when build flags change without # any pin changing, so venvs that already carry .ready are not reused with a stale recipe. -COLLX_DEEPEP_V2_BUILD_GEN="dlarch2" +COLLX_DEEPEP_V2_BUILD_GEN="imgstack1" COLLX_UCCL_REPO="https://github.com/uccl-project/uccl" COLLX_UCCL_COMMIT="df000fd3001da04a397253fd6a2204bde62d32aa" diff --git a/collectivex/runtime/prepare_backend.sh b/collectivex/runtime/prepare_backend.sh index ee30c69c07..ab5391b042 100644 --- a/collectivex/runtime/prepare_backend.sh +++ b/collectivex/runtime/prepare_backend.sh @@ -198,12 +198,14 @@ deepep_ready() { [ -f "$1/.ready" ] && [ -x "$1/venv/bin/python" ] && [ -d "$1/s deepep_cuda_header_overlay() { local root="$1" cuda_home="$2" overlay="$1/cuda-include" wheel_include header mkdir -p "$overlay" || return 1 - for wheel_include in "$root"/venv/lib/python*/site-packages/nvidia/cu13/include; do + local nvidia_dir + while IFS= read -r nvidia_dir; do + wheel_include="$nvidia_dir/cu13/include" [ -d "$wheel_include" ] || continue for header in "$wheel_include"/*.h; do [ -e "$cuda_home/include/${header##*/}" ] || ln -sfn "$header" "$overlay/" || return 1 done - done + done < <("$root/venv/bin/python" -c 'import nvidia; print("\n".join(nvidia.__path__))') } deepep_activate() { @@ -274,19 +276,25 @@ deepep_install() { fi mkdir -m 700 "$root" \ || { collx_log "ERROR: DeepEP V2 cache-create failed"; return 1; } - python3 -m venv "$venv" \ + # "image" builds against the image's own torch and NCCL, as vLLM's installer does. + local -a venv_flags=() tools=("pip==26.1.2" "setuptools==82.0.1" "wheel==0.47.0" "ninja==1.13.0") + if [ "$COLLX_DEEPEP_V2_TORCH_SPEC" = image ]; then venv_flags=(--system-site-packages) + else tools+=("numpy==2.2.6"); fi + python3 -m venv "${venv_flags[@]}" "$venv" \ || { collx_log "ERROR: DeepEP V2 venv creation failed"; return 1; } pip=("$venv/bin/python" -m pip install -q --disable-pip-version-check --no-input) - "${pip[@]}" \ - "pip==26.1.2" "setuptools==82.0.1" "wheel==0.47.0" "ninja==1.13.0" \ - "numpy==2.2.6" "$COLLX_DEEPEP_V2_NVSHMEM_SPEC" >&2 2>&1 \ + "${pip[@]}" "${tools[@]}" "$COLLX_DEEPEP_V2_NVSHMEM_SPEC" >&2 2>&1 \ || { collx_log "ERROR: DeepEP V2 build-tool installation failed"; return 1; } - "${pip[@]}" --index-url https://download.pytorch.org/whl/cu130 \ - --extra-index-url https://pypi.org/simple "$COLLX_DEEPEP_V2_TORCH_SPEC" >&2 2>&1 \ - || { collx_log "ERROR: $COLLX_DEEPEP_V2_TORCH_SPEC (cu130) installation failed"; return 1; } - # Torch pins NCCL 2.28.9; see COLLX_DEEPEP_V2_NCCL_SPEC for what each source needs. - "${pip[@]}" --force-reinstall --no-deps "$COLLX_DEEPEP_V2_NCCL_SPEC" >&2 2>&1 \ - || { collx_log "ERROR: $COLLX_DEEPEP_V2_NCCL_SPEC installation failed"; return 1; } + if [ "$COLLX_DEEPEP_V2_TORCH_SPEC" != image ]; then + "${pip[@]}" --index-url https://download.pytorch.org/whl/cu130 \ + --extra-index-url https://pypi.org/simple "$COLLX_DEEPEP_V2_TORCH_SPEC" >&2 2>&1 \ + || { collx_log "ERROR: $COLLX_DEEPEP_V2_TORCH_SPEC (cu130) installation failed"; return 1; } + fi + # Torch pins an older NCCL than ElasticBuffer needs; the vLLM image already carries >= 2.30.4. + if [ "$COLLX_DEEPEP_V2_NCCL_SPEC" != image ]; then + "${pip[@]}" --force-reinstall --no-deps "$COLLX_DEEPEP_V2_NCCL_SPEC" >&2 2>&1 \ + || { collx_log "ERROR: $COLLX_DEEPEP_V2_NCCL_SPEC installation failed"; return 1; } + fi local toolchain cuda_home cccl toolchain="$(cuda_toolchain_paths)" || return 1 IFS=$'\t' read -r cuda_home cccl <<< "$toolchain" From 04118aa7f843919f095a127e58e682ce1ec4685f Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Sat, 7 Nov 2026 01:06:39 -0800 Subject: [PATCH 027/111] =?UTF-8?q?[VideoGenX]=20score=20sweep=20clips=20a?= =?UTF-8?q?gainst=20Diffusers=20reference=20videos=20/=20[VideoGenX]=20?= =?UTF-8?q?=E5=B0=86=20sweep=20=E8=A7=86=E9=A2=91=E4=B8=8E=20Diffusers=20?= =?UTF-8?q?=E5=8F=82=E8=80=83=E8=A7=86=E9=A2=91=E5=AF=B9=E6=AF=94=E6=89=93?= =?UTF-8?q?=E5=88=86=20(#4659)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: verify GPU ownership and seal H3 execution evidence * feat: define reproducible H3 workload sets * feat: measure H3 requests and validate generated media * refactor: delete unused OwnedProcess.owns helper * refactor: fold H3 spec digest and case helpers * test: table-drive H3 spec rejection cases * chore: compact H3 spec YAML and drop stale references * fix: use warmup case in GPU report slots * refactor: drop unreachable GPU job guards * test: drop dead parametrize branches in measurement tests * refactor: share no-follow open and canonical JSON helpers * refactor: share deadline budget in GPU probes * refactor: reuse runner slots and hashing in compare * test: drop dead parametrize branches in GPU tests * refactor: dedupe evaluator encoding, status, and quantile rules * refactor: move offline compare and report tools out of measurement core * chore: drop parked offline GPU tools superseded by native sweeps * feat: add public H3 reference descriptor * feat: score h3-results clips against public references * ci: score H3 sweep artifacts against the reference release * docs: document H3 reference PSNR scoring * fix: score each deployment's newest bundle and survive unreadable ones --- .github/workflows/video-quality.yml | 106 +++++++++ video-generation/README.md | 27 +++ video-generation/README_zh.md | 24 ++ video-generation/evaluator/cli.py | 41 ++++ video-generation/evaluator/quality.py | 226 +++++++++++++++++++ video-generation/reference/h3-v2.json | 292 +++++++++++++++++++++++++ video-generation/tests/test_quality.py | 179 +++++++++++++++ 7 files changed, 895 insertions(+) create mode 100644 .github/workflows/video-quality.yml create mode 100644 video-generation/evaluator/quality.py create mode 100644 video-generation/reference/h3-v2.json create mode 100644 video-generation/tests/test_quality.py diff --git a/.github/workflows/video-quality.yml b/.github/workflows/video-quality.yml new file mode 100644 index 0000000000..7f2e90f82f --- /dev/null +++ b/.github/workflows/video-quality.yml @@ -0,0 +1,106 @@ +name: Video Quality +run-name: >- + Video quality - run ${{ inputs.run_id || github.event.workflow_run.id }} + attempt ${{ inputs.run_attempt || github.event.workflow_run.run_attempt }} + +on: + workflow_run: # zizmor: ignore[dangerous-triggers] reads the triggering run's artifacts only as data + workflows: [Run Sweep] + types: [completed] + workflow_dispatch: + inputs: + run_id: + description: Run whose h3-results artifacts to score + type: string + required: true + run_attempt: + description: Attempt that names the quality artifact + type: string + required: true + +permissions: {} + +concurrency: + group: video-quality-${{ inputs.run_id || github.event.workflow_run.id }}-${{ inputs.run_attempt || github.event.workflow_run.run_attempt }} + cancel-in-progress: false + +env: + RUN_ID: ${{ inputs.run_id || github.event.workflow_run.id }} + RUN_ATTEMPT: ${{ inputs.run_attempt || github.event.workflow_run.run_attempt }} + +jobs: + find: + name: Find h3-results artifacts + # Fork runs are scored only by an explicit dispatch. + if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.head_repository.full_name == github.repository + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + actions: read # List the run's artifacts. + outputs: + artifacts: ${{ steps.list.outputs.artifacts }} + steps: + - name: List the newest h3-results bundle of each deployment + id: list + env: + GH_TOKEN: ${{ github.token }} + run: | + set -eo pipefail + [[ "$RUN_ID" =~ ^[0-9]+$ && "$RUN_ATTEMPT" =~ ^[0-9]+$ ]] || { + echo "::error::run_id and run_attempt must be numeric" + exit 1 + } + # Names are h3-results---; a rerun's bundle replaces the + # earlier attempt's, and a bundle from a later attempt is never scored. + artifacts=$(gh api --paginate "repos/$GITHUB_REPOSITORY/actions/runs/$RUN_ID/artifacts" \ + --jq '.artifacts[] | select((.expired | not) and (.name | startswith("h3-results-"))) | {id, name}' \ + | jq -sc --arg run "$RUN_ID" --argjson attempt "$RUN_ATTEMPT" ' + [.[] | (.name | split("-")) as $p + | select(($p | length) == 5 and $p[2] == $run and ($p[3] | tonumber) <= $attempt) + | {id, attempt: ($p[3] | tonumber), fingerprint: $p[4]}] + | group_by(.fingerprint) | map(max_by(.attempt).id) | sort') + echo "artifacts=$artifacts" >> "$GITHUB_OUTPUT" + [[ "$artifacts" != "[]" ]] || echo "Run $RUN_ID has no h3-results artifacts; nothing to score." + + score: + name: Score H3 clips against references + needs: find + if: needs.find.outputs.artifacts != '[]' + runs-on: ubuntu-latest + timeout-minutes: 120 + permissions: + actions: read # Download the run's h3-results artifacts. + contents: read # Check out main and download the reference release. + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + - name: Download references and result bundles + env: + GH_TOKEN: ${{ github.token }} + ARTIFACTS: ${{ needs.find.outputs.artifacts }} + run: | + tag=$(jq -r .release.tag video-generation/reference/h3-v2.json) + gh release download "$tag" --repo "$GITHUB_REPOSITORY" --pattern '*.mp4' \ + --dir "$RUNNER_TEMP/references" + mkdir "$RUNNER_TEMP/bundles" + for id in $(jq -r '.[]' <<< "$ARTIFACTS"); do + gh api "repos/$GITHUB_REPOSITORY/actions/artifacts/$id/zip" > "$RUNNER_TEMP/bundles/$id.zip" + done + - name: Score every measured clip + working-directory: video-generation + run: | + bundles=() + for zip in "$RUNNER_TEMP"/bundles/*.zip; do + bundles+=("$(basename "$zip" .zip)=$zip") + done + uv run --no-project --python 3.12 --with 'av==16.1.0' --with 'numpy==2.3.5' \ + python -m evaluator.cli score-quality "${bundles[@]}" \ + --descriptor reference/h3-v2.json --references "$RUNNER_TEMP/references" \ + --run-id "$RUN_ID" --source-revision "$GITHUB_SHA" --output "$RUNNER_TEMP/h3-quality" + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: h3-quality-${{ env.RUN_ID }}-${{ env.RUN_ATTEMPT }} + path: ${{ runner.temp }}/h3-quality + if-no-files-found: error diff --git a/video-generation/README.md b/video-generation/README.md index cb40d4250a..13e2f79eb4 100644 --- a/video-generation/README.md +++ b/video-generation/README.md @@ -53,6 +53,33 @@ separate evidence. Comparisons leave cross-engine quality unjudged, while same-implementation numerical checks require matching sealed image, launch, environment, GPU and algorithm identity. +## Reference PSNR + +[Video Quality](../.github/workflows/video-quality.yml) scores every measured clip +of a finished Run Sweep against the Diffusers references. It runs main's code on a +GitHub-hosted runner and reads the sweep's `h3-results-*` artifacts only as data. +[`reference/h3-v2.json`](reference/h3-v2.json) names the reference release, the +generation settings and implementation identity, and each case's prompt, seed, +MP4 SHA256 and size. The job downloads the release assets, rejects any asset whose +hash differs from the descriptor, and runs: + +```bash +python -m evaluator.cli score-quality = ... \ + --descriptor reference/h3-v2.json --references \ + --run-id --source-revision --output +``` + +A clip is scored only when its recorded prompt, seed, model and generation +settings equal the descriptor; `compare_media` then compares every decoded frame. +Other clips are listed under `unscored` with a reason: `invalid_candidate`, +`checksum_mismatch`, `no_reference`, or `settings_mismatch` with the differing +fields. A bundle that cannot be read is listed under `unreadable` and the others are still +scored. Each deployment contributes its newest bundle up to the scored attempt. The +`h3-quality--` artifact holds `quality.json`, a copy of +the descriptor and `SHA256SUMS`. PSNR measures technical fidelity to one reference +implementation, so `quality_status` stays `unjudged`. To score an earlier run, +dispatch the workflow with its `run_id` and `run_attempt`. + ## Local checks ```bash diff --git a/video-generation/README_zh.md b/video-generation/README_zh.md index 1e12bbefd4..a82321faa0 100644 --- a/video-generation/README_zh.md +++ b/video-generation/README_zh.md @@ -46,6 +46,30 @@ prompt 合同，集合选择器为 `release-v1`。请求使用 49 次去噪更 实际编码器实现和跨引擎质量仍需单独证据。比较器将跨引擎质量标为未评估； 同实现数值检查则要求封存的镜像、启动命令、环境、GPU 和算法身份一致。 +## 参考 PSNR + +[Video Quality](../.github/workflows/video-quality.yml) 在 Run Sweep 结束后， +以 Diffusers 参考视频为基准，为每段测量视频计算 PSNR。它在 GitHub 托管的 +runner 上运行 main 分支的代码，sweep 的 `h3-results-*` 产物只作为数据读取。 +[`reference/h3-v2.json`](reference/h3-v2.json) 记录参考视频所在的 release、 +生成参数与实现身份，以及每个 case 的 prompt、seed、MP4 的 SHA256 和字节数。 +任务先下载 release 资源，哈希与描述文件不一致的资源直接拒绝，然后运行： + +```bash +python -m evaluator.cli score-quality = ... \ + --descriptor reference/h3-v2.json --references \ + --run-id --source-revision --output +``` + +只有记录的 prompt、seed、模型与生成参数都与描述文件一致时才评分，随后由 +`compare_media` 比较全部解码帧。其余视频列在 `unscored` 中并注明原因： +`invalid_candidate`、`checksum_mismatch`、`no_reference`，或 `settings_mismatch` +及不一致的字段。无法读取的 bundle 列在 `unreadable` 中，其余 bundle 照常评分。 +每个部署只取不晚于该 attempt 的最新 bundle。`h3-quality--` 产物包含 `quality.json`、描述文件 +副本和 `SHA256SUMS`。PSNR 只衡量相对单一参考实现的技术保真度，因此 +`quality_status` 仍为 `unjudged`。如需为较早的运行补算，按其 `run_id` 与 +`run_attempt` 手动触发该 workflow。 + ## 本地检查 ```bash diff --git a/video-generation/evaluator/cli.py b/video-generation/evaluator/cli.py index d510902792..cbe201816a 100644 --- a/video-generation/evaluator/cli.py +++ b/video-generation/evaluator/cli.py @@ -133,6 +133,23 @@ def interrupt_run(_signum, _frame): print(json.dumps(result, indent=2, sort_keys=True, allow_nan=False)) return {"pass": 0, "fail": 1, "inconclusive": 2}[result["overall_status"]] + if arguments.command == "score-quality": + from .quality import score_quality + + bundles = dict(arguments.bundles) + if len(bundles) != len(arguments.bundles): + raise ValueError("each artifact ID may appear only once") + result = score_quality( + bundles=bundles, + references=arguments.references, + descriptor=arguments.descriptor, + run_id=arguments.run_id, + source_revision=arguments.source_revision, + output=arguments.output, + ) + print(json.dumps(result, indent=2, sort_keys=True, allow_nan=False)) + return 0 + if arguments.command == "inspect-media": from .media import analyze_media @@ -149,6 +166,13 @@ def interrupt_run(_signum, _frame): raise AssertionError(arguments.command) +def _bundle_argument(value: str) -> tuple[int, Path]: + artifact, separator, path = value.partition("=") + if not separator or not artifact.isdigit() or not path: + raise argparse.ArgumentTypeError("expected ARTIFACT_ID=PATH") + return int(artifact), Path(path) + + def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="vgbench") subparsers = parser.add_subparsers(dest="command", required=True) @@ -210,6 +234,23 @@ def _parser() -> argparse.ArgumentParser: "--report", type=Path, help="new portable HTML report plus a local media-assets directory" ) + score = subparsers.add_parser( + "score-quality", + help="score stored h3-results clips against the public Diffusers reference release", + ) + score.add_argument( + "bundles", + nargs="+", + type=_bundle_argument, + metavar="ARTIFACT_ID=PATH", + help="h3-results bundle, as a zip or directory, and its GitHub artifact ID", + ) + score.add_argument("--descriptor", required=True, type=Path) + score.add_argument("--references", required=True, type=Path, help="directory of release MP4s") + score.add_argument("--run-id", required=True, help="GitHub run that stores the bundles") + score.add_argument("--source-revision", required=True, help="git commit of this scorer") + score.add_argument("--output", required=True, type=Path, help="new attachment directory") + inspect = subparsers.add_parser( "inspect-media", help="fully decode a file and measure technical media integrity" ) diff --git a/video-generation/evaluator/quality.py b/video-generation/evaluator/quality.py new file mode 100644 index 0000000000..a935d736c7 --- /dev/null +++ b/video-generation/evaluator/quality.py @@ -0,0 +1,226 @@ +from __future__ import annotations + +import hashlib +import json +import re +import tempfile +import zipfile +from collections import Counter +from pathlib import Path, PurePosixPath +from typing import Any, Iterator + +from . import media + +PLAN_FIELDS = ("model_id", "model_revision", "generation", "fairness") +CASE_FIELDS = ("case_id", "prompt", "seed", "requires_motion", "requires_sound") + + +def _digest(path: Path) -> str: + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def _checksums(root: Path) -> dict[str, str]: + sums: dict[str, str] = {} + for line in (root / "SHA256SUMS").read_text().splitlines(): + match = re.fullmatch(r"([0-9a-f]{64}) (.+)", line) + if not match or match[2] in sums: + raise ValueError(f"{root.name}: malformed or duplicate checksum entry") + sums[match[2]] = match[1] + return sums + + +def _references(descriptor: dict, directory: Path) -> dict[str, tuple[dict, Path]]: + references = {} + for case in descriptor["cases"]: + path = directory / case["asset"] + if ( + not path.is_file() + or path.stat().st_size != case["bytes"] + or _digest(path) != case["sha256"] + ): + raise ValueError(f"reference asset {case['asset']} does not match the descriptor") + references[case["case_id"]] = (case, path) + return references + + +def _measured( + root: Path, sums: dict[str, str], run_id: str, artifact_id: int +) -> Iterator[tuple[dict, dict, dict]]: + manifest_sha = _digest(root / "manifest.json") + if sums.get("manifest.json") != manifest_sha: + raise ValueError(f"artifact {artifact_id}: manifest.json does not match SHA256SUMS") + source_id = json.loads((root / "manifest.json").read_bytes())["run_id"] + for cell in json.loads((root / "serving-smoke.json").read_bytes())["cells"]: + cell_id = f"c{cell['concurrency']}" + base = PurePosixPath("gpu", cell_id, "baseline") + if not (root / base / "run.json").is_file(): + continue + run = json.loads((root / base / "run.json").read_bytes()) + for record in run.get("records") or []: + if record.get("phase") != "measurement": + continue + relative = PurePosixPath(record.get("artifact_path") or "") + safe = relative.parts and not relative.is_absolute() and ".." not in relative.parts + yield ( + run["plan"], + record, + { + "run_id": run_id, + "artifact_id": artifact_id, + "source_id": source_id, + "manifest_sha256": manifest_sha, + "cell_id": cell_id, + "slot_id": record.get("slot_id"), + "case_id": record.get("case_id"), + "prompt": record.get("prompt"), + "seed": record.get("seed"), + "media_path": (base / relative).as_posix() if safe else None, + "media_sha256": record.get("sha256"), + }, + ) + + +def _unscored_reason( + root: Path, + sums: dict[str, str], + plan: dict, + record: dict, + candidate: dict, + descriptor: dict, + references: dict[str, tuple[dict, Path]], +) -> tuple[str, list[str]] | None: + path = candidate["media_path"] + if ( + record.get("status") != "succeeded" + or (record.get("media") or {}).get("valid") is not True + or path is None + or not (root / path).is_file() + ): + return "invalid_candidate", [] + if not (candidate["media_sha256"] == sums.get(path) == _digest(root / path)): + return "checksum_mismatch", [] + if candidate["case_id"] not in references: + return "no_reference", [] + reference = references[candidate["case_id"]][0] + planned = [ + case for case in plan.get("cases") or [] if case.get("case_id") == candidate["case_id"] + ] + recorded = { + **(planned[0] if len(planned) == 1 else {}), + **{field: candidate[field] for field in ("case_id", "prompt", "seed")}, + } + fields = [field for field in PLAN_FIELDS if plan.get(field) != descriptor[field]] + fields += [field for field in CASE_FIELDS if recorded.get(field) != reference[field]] + return ("settings_mismatch", fields) if fields else None + + +def _error(error: Exception) -> str: + if isinstance(error, OSError) and error.filename: + return f"{type(error).__name__}: {Path(error.filename).name}" + return f"{type(error).__name__}: {error}" + + +def _score_bundle( + artifact_id: int, + path: Path, + scratch: Path, + run_id: str, + descriptor: dict, + references: dict[str, tuple[dict, Path]], +) -> tuple[list[dict], list[dict]]: + root = path + if path.is_file(): + root = scratch / str(artifact_id) + with zipfile.ZipFile(path) as archive: + archive.extractall(root) + sums = _checksums(root) + pairs, unscored = [], [] + for plan, record, candidate in _measured(root, sums, run_id, artifact_id): + failure = _unscored_reason(root, sums, plan, record, candidate, descriptor, references) + if failure is not None: + reason, fields = failure + unscored.append({"candidate": candidate, "reason": reason, "fields": fields}) + continue + case, reference = references[candidate["case_id"]] + pairs.append( + { + "candidate": candidate, + "reference": { + "receipt_sha256": case["receipt_sha256"], + "case_id": case["case_id"], + "media_path": case["asset"], + "media_sha256": case["sha256"], + }, + "comparison": media.compare_media(reference, root / candidate["media_path"]), + } + ) + return pairs, unscored + + +def score_quality( + *, + bundles: dict[int, Path], + references: Path, + descriptor: Path, + run_id: str, + source_revision: str, + output: Path, +) -> dict[str, Any]: + """Score every measured clip against its public reference; unscorable clips and bundles stay listed.""" + if not re.fullmatch(r"[0-9a-f]{40}", source_revision): + raise ValueError("source revision must be a 40-character git commit") + if output.exists(): + raise ValueError(f"output already exists: {output}") + raw = descriptor.read_bytes() + spec = json.loads(raw) + known = _references(spec, references) + pairs, unscored, unreadable = [], [], [] + with tempfile.TemporaryDirectory() as scratch: + for artifact_id, path in sorted(bundles.items()): + try: + scored, skipped = _score_bundle( + artifact_id, path, Path(scratch), run_id, spec, known + ) + except (OSError, ValueError, KeyError, TypeError, zipfile.BadZipFile) as error: + unreadable.append({"artifact_id": artifact_id, "error": _error(error)}) + continue + pairs += scored + unscored += skipped + attachment = { + "schema_version": 1, + "quality_status": "unjudged", + "release_qualified": False, + "scorer": { + "name": "evaluator.media.compare_media", + "method": "decoded-rgb24-all-frames", + "source_revision": source_revision, + "sha256": _digest(Path(media.__file__)), + }, + "references": { + "descriptor": descriptor.name, + "descriptor_sha256": hashlib.sha256(raw).hexdigest(), + "release": spec["release"], + }, + "pairs": pairs, + "unscored": unscored, + "unreadable": unreadable, + } + output.mkdir(parents=True) + (output / descriptor.name).write_bytes(raw) + (output / "quality.json").write_text( + json.dumps(attachment, indent=2, sort_keys=True, allow_nan=False) + "\n" + ) + (output / "SHA256SUMS").write_text( + "".join( + f"{_digest(output / name)} {name}\n" + for name in sorted([descriptor.name, "quality.json"]) + ) + ) + return { + "output": str(output), + "pairs": len(pairs), + "compatible": sum(pair["comparison"]["compatible"] for pair in pairs), + "unscored": dict(sorted(Counter(item["reason"] for item in unscored).items())), + "unreadable": len(unreadable), + } diff --git a/video-generation/reference/h3-v2.json b/video-generation/reference/h3-v2.json new file mode 100644 index 0000000000..d53865aa38 --- /dev/null +++ b/video-generation/reference/h3-v2.json @@ -0,0 +1,292 @@ +{ + "schema_version": 1, + "spec": "specs/h3-v2.yaml", + "set": "release-v1", + "release": { + "repository": "SemiAnalysisAI/InferenceX", + "tag": "h3-v2-release-v1-references" + }, + "requirements_lock_sha256": "0e79ab689f0b5285b0cd2a5b4dbb26fce396eb8fdbf599046e5a479cca675988", + "model_id": "MiniMaxAI/MiniMax-H3", + "model_revision": "42ed227ee7df40d41602854ae760620d6eb651fe", + "generation": { + "aspect_ratio": "16:9", + "audio_channels": 2, + "audio_flow_shift": 3.0, + "audio_sample_rate_hz": 32000, + "duration_seconds": 5, + "flow_shift": 12.0, + "fps": 24, + "frame_count": 124, + "height": 768, + "num_inference_steps": 49, + "width": 1344 + }, + "fairness": { + "schedule": { + "sigma_points": 50, + "denoising_updates": 49, + "video_flow_shift": 12.0, + "audio_flow_shift": 3.0 + }, + "encoding": { + "video_codec": "libx264", + "preset": "ultrafast", + "crf": 18, + "pixel_format": "yuv420p", + "audio_codec": "aac", + "audio_sample_rate_hz": 32000, + "audio_channels": 2 + }, + "conditioning_cache": "disabled_across_requests", + "reference_identity": { + "implementation": "official-diffusers-t2va", + "minimax_code_revision": "199b7d0e1fd95e7b3eafa8c15eca692534142427", + "diffusers_revision": "d8c1b13c756df0059bcdb6e477a1eafcd502098d", + "model_revision": "42ed227ee7df40d41602854ae760620d6eb651fe", + "rng_policy": "cpu_generator_float32_video_then_audio" + } + }, + "cases": [ + { + "case_id": "ceramic-wheel", + "prompt": "A single continuous shot of a potter shaping a wet clay bowl on a spinning wheel. The action is visible and the wheel humming and wet hands brushing clay is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 11, + "requires_motion": true, + "requires_sound": true, + "asset": "ceramic-wheel.mp4", + "sha256": "a4ac6f7f3679810e8c9c8533eaddec4d6a11e5a20010ca63931c2035929768aa", + "bytes": 6524325, + "receipt_sha256": "e1b67fe6c603c3e9989bcb77186b27eb71d589bbe58a1cccca8d06b96a11af72", + "plan_sha256": "5367da02081a442259f29cc224f99a43171c7436842a915d098eb11110f19ec7" + }, + { + "case_id": "paper-fold", + "prompt": "A single continuous shot of hands folding a blue sheet into a paper crane. The action is visible and the paper creasing is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 23, + "requires_motion": true, + "requires_sound": true, + "asset": "paper-fold.mp4", + "sha256": "6b4b38d57d35f927585b1589c07fc999a7b8d7552d02a7dc5e68e16d92a6ffe8", + "bytes": 6273811, + "receipt_sha256": "c19249ae7a8e58e3d6a13427d302d51f3b939f63c9ce92c03bf91a9ad07b504b", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "wind-chimes", + "prompt": "A single continuous shot of three metal wind chimes swinging on a porch. The action is visible and the chimes ringing is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 29, + "requires_motion": true, + "requires_sound": true, + "asset": "wind-chimes.mp4", + "sha256": "0a495d81498618f7bfb6e3c3c54a096a407c2c447119ff2182f7fa4c19488dd7", + "bytes": 2268482, + "receipt_sha256": "70013af0bc288ccf2f654c24192b93c72e4b0e0ab3c662a3fe3d4f9cf809a1e8", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "coffee-grinder", + "prompt": "A single continuous shot of a manual coffee grinder turning beside a cup. The action is visible and the grinder rattling is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 47, + "requires_motion": true, + "requires_sound": true, + "asset": "coffee-grinder.mp4", + "sha256": "1981ac0affbdbc9a468243f060cd47a37933e2c0401025208ca7f60985247902", + "bytes": 3117900, + "receipt_sha256": "b84002bd21555e37dc5d352929aed2d6530ccba3ba3e418d982489570ed927fc", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "dog-shake", + "prompt": "A single continuous shot of a wet dog shaking water from its fur in a yard. The action is visible and the collar jingling is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 83, + "requires_motion": true, + "requires_sound": true, + "asset": "dog-shake.mp4", + "sha256": "b8498affae61218c2a6e3a3e247fb52ce59d2f2c804110d069d01ca78b56dd0d", + "bytes": 6942495, + "receipt_sha256": "84e22ba809cbffd675588cb154d8a65595ddef181a8a486bbbf46f32796c8517", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "zipper-close", + "prompt": "A single continuous shot of a hand closing a zipper on a red canvas bag. The action is visible and the zipper sliding is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 11, + "requires_motion": true, + "requires_sound": true, + "asset": "zipper-close.mp4", + "sha256": "360d107e12e5ad7100879436ebf6d77bdd52f32a195239a0c274f24fa2536f94", + "bytes": 9307366, + "receipt_sha256": "0da87381bab9cce250f41447b66d50c035ac0f70830b4afc26f9c8b8a378e8db", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "rolling-ball", + "prompt": "A single continuous shot of a glass ball rolling across a wooden table and stopping at a book. The action is visible and the ball tapping the book is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 23, + "requires_motion": true, + "requires_sound": true, + "asset": "rolling-ball.mp4", + "sha256": "489adada3e8d917a7e61ed15f48b0dd5f59f721e9b7f542a7e32279540812d5e", + "bytes": 2096816, + "receipt_sha256": "281e9d8f742b82466980b2cab267e1aec6910d968bdfa59dd160834676efadee", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "paint-brush", + "prompt": "A single continuous shot of a brush painting a yellow stroke across white paper. The action is visible and the brush rubbing the paper is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 29, + "requires_motion": true, + "requires_sound": true, + "asset": "paint-brush.mp4", + "sha256": "bb3e58bd2d0be83b044438f2abb994e04aa09bcff4594bfb13c9e7dd1de7292b", + "bytes": 4608981, + "receipt_sha256": "c19249ae7a8e58e3d6a13427d302d51f3b939f63c9ce92c03bf91a9ad07b504b", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "kettle-pour", + "prompt": "A single continuous shot of a kettle pouring hot water into a mug. The action is visible and the water splashing is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 47, + "requires_motion": true, + "requires_sound": true, + "asset": "kettle-pour.mp4", + "sha256": "768e016c9c72b642367fba490d4c5a0f1192aa787a46823509a2ba1802216773", + "bytes": 2355142, + "receipt_sha256": "70013af0bc288ccf2f654c24192b93c72e4b0e0ab3c662a3fe3d4f9cf809a1e8", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "candle-light", + "prompt": "A single continuous shot of a match lighting a candle wick. The action is visible and the match striking is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 83, + "requires_motion": true, + "requires_sound": true, + "asset": "candle-light.mp4", + "sha256": "9f20d3ec52b24818e931ff9c57a35bff361a1d781c2eeb0095111bb4a2a34443", + "bytes": 2733338, + "receipt_sha256": "b84002bd21555e37dc5d352929aed2d6530ccba3ba3e418d982489570ed927fc", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "book-pages", + "prompt": "A single continuous shot of a hand turning three pages of a book. The action is visible and the pages rustling is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 11, + "requires_motion": true, + "requires_sound": true, + "asset": "book-pages.mp4", + "sha256": "af5272bb0aaa4989e0ada1a8321b8f5b6b5192b719bc0729da428ef4765e6d2d", + "bytes": 6493490, + "receipt_sha256": "84e22ba809cbffd675588cb154d8a65595ddef181a8a486bbbf46f32796c8517", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "basketball-bounce", + "prompt": "A single continuous shot of a basketball bouncing three times on an empty court. The action is visible and each ball bounce is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 23, + "requires_motion": true, + "requires_sound": true, + "asset": "basketball-bounce.mp4", + "sha256": "20f5b88911fb5c8ac64bdc312f1bcc667f6d706be0663471b9cab0904bc8a1e0", + "bytes": 3008554, + "receipt_sha256": "0da87381bab9cce250f41447b66d50c035ac0f70830b4afc26f9c8b8a378e8db", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "window-rain", + "prompt": "A single continuous shot of raindrops sliding down a window. The action is visible and the light rain pattering is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 29, + "requires_motion": true, + "requires_sound": true, + "asset": "window-rain.mp4", + "sha256": "a21af9d365445264a207590e3af8adeb23b549961868217eb14dd23288254e80", + "bytes": 2245929, + "receipt_sha256": "281e9d8f742b82466980b2cab267e1aec6910d968bdfa59dd160834676efadee", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "leaves-rake", + "prompt": "A single continuous shot of a gardener raking dry leaves into a pile. The action is visible and the leaves scraping is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 47, + "requires_motion": true, + "requires_sound": true, + "asset": "leaves-rake.mp4", + "sha256": "4e9f84c7ea79db5c1ce6458ebaf98f56a189edaf4475264aab9609451aaec416", + "bytes": 8139094, + "receipt_sha256": "c19249ae7a8e58e3d6a13427d302d51f3b939f63c9ce92c03bf91a9ad07b504b", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "sewing-machine", + "prompt": "A single continuous shot of a sewing machine stitching blue fabric. The action is visible and the motor cycling is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 83, + "requires_motion": true, + "requires_sound": true, + "asset": "sewing-machine.mp4", + "sha256": "c07bdf97a2e113eb43daa4f0591a6bc9f6f4b007e5f615178efc7efa90ece7e9", + "bytes": 5808216, + "receipt_sha256": "70013af0bc288ccf2f654c24192b93c72e4b0e0ab3c662a3fe3d4f9cf809a1e8", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "ping-pong", + "prompt": "A single continuous shot of two players exchanging three ping-pong hits. The action is visible and each ball contact is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 11, + "requires_motion": true, + "requires_sound": true, + "asset": "ping-pong.mp4", + "sha256": "cd77ad512940a53d3754dc4a72e24f9c2dc61750f56f3aac5ad9a0c9b21c05ab", + "bytes": 7919977, + "receipt_sha256": "b84002bd21555e37dc5d352929aed2d6530ccba3ba3e418d982489570ed927fc", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "umbrella-open", + "prompt": "A single continuous shot of a person opening a folded umbrella on a dry patio. The action is visible and the release clicking is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 23, + "requires_motion": true, + "requires_sound": true, + "asset": "umbrella-open.mp4", + "sha256": "586bcde9312aa3c08182857193a49bd6b69e55cf0fb5a4ef680b6f2610972781", + "bytes": 4562676, + "receipt_sha256": "84e22ba809cbffd675588cb154d8a65595ddef181a8a486bbbf46f32796c8517", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "shoe-laces", + "prompt": "A single continuous shot of hands tying the laces of a running shoe. The action is visible and the laces rubbing is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 29, + "requires_motion": true, + "requires_sound": true, + "asset": "shoe-laces.mp4", + "sha256": "ff75ad4932b34f8c3f6cf6c53611dca90fac802373b633e196c0dd461d57ca73", + "bytes": 6691641, + "receipt_sha256": "0da87381bab9cce250f41447b66d50c035ac0f70830b4afc26f9c8b8a378e8db", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "ice-cubes", + "prompt": "A single continuous shot of three ice cubes dropping into a clear glass. The action is visible and the cubes clinking is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 47, + "requires_motion": true, + "requires_sound": true, + "asset": "ice-cubes.mp4", + "sha256": "e5fad0b9023eb0bed3e7670fa156e1858116a0db7c695863cb16be3301f8a3b5", + "bytes": 3997428, + "receipt_sha256": "281e9d8f742b82466980b2cab267e1aec6910d968bdfa59dd160834676efadee", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + }, + { + "case_id": "station-clock", + "prompt": "A single continuous shot of an analog clock with a moving second hand. The action is visible and the clock ticking is audible. Fixed camera, no speech, music, subtitles, or cuts.", + "seed": 83, + "requires_motion": false, + "requires_sound": true, + "asset": "station-clock.mp4", + "sha256": "3e6361b1cd3f665d83bae129098ea6ecdddb72a1cd0e65f05b115bb496059554", + "bytes": 2492095, + "receipt_sha256": "c19249ae7a8e58e3d6a13427d302d51f3b939f63c9ce92c03bf91a9ad07b504b", + "plan_sha256": "fbfa03afd18e998e1e488aa3417a0bce09dc15e36a427eb578fced0444a99c34" + } + ] +} diff --git a/video-generation/tests/test_quality.py b/video-generation/tests/test_quality.py new file mode 100644 index 0000000000..86accb6ec1 --- /dev/null +++ b/video-generation/tests/test_quality.py @@ -0,0 +1,179 @@ +import hashlib +import json +import shutil +from pathlib import Path + +import pytest + +from evaluator import cli +from test_media import encode_media + +REVISION = "a" * 40 +CASES = [ + { + "case_id": name, + "prompt": f"{name} prompt", + "seed": seed, + "requires_motion": True, + "requires_sound": True, + } + for seed, name in enumerate(("same", "changed", "reseeded"), 1) +] +SETTINGS = { + "model_id": "MiniMaxAI/MiniMax-H3", + "model_revision": "b" * 40, + "generation": {"frame_count": 8, "fps": 10}, + "fairness": {"encoding": {"crf": 18}}, +} + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _setup(root: Path) -> tuple[Path, Path, Path]: + references = root / "references" + references.mkdir() + for case in CASES: + encode_media(references / f"{case['case_id']}.mp4") + descriptor = root / "h3-v2.json" + descriptor.write_text( + json.dumps( + { + **SETTINGS, + "release": {"repository": "SemiAnalysisAI/InferenceX", "tag": "refs-v1"}, + "cases": [ + { + **case, + "asset": f"{case['case_id']}.mp4", + "sha256": _sha(references / f"{case['case_id']}.mp4"), + "bytes": (references / f"{case['case_id']}.mp4").stat().st_size, + "receipt_sha256": "c" * 64, + "plan_sha256": "d" * 64, + } + for case in CASES + ], + } + ) + ) + bundle = root / "bundle" + artifacts = bundle / "gpu/c1/baseline/artifacts" + artifacts.mkdir(parents=True) + shutil.copyfile(references / "same.mp4", artifacts / "measurement-r001-c001.mp4") + encode_media(artifacts / "measurement-r001-c002.mp4", video_change=True) + shutil.copyfile(references / "reseeded.mp4", artifacts / "measurement-r001-c003.mp4") + records = [ + { + "phase": "measurement", + "slot_id": f"measurement-r001-c{index:03d}", + "case_id": case["case_id"], + "prompt": case["prompt"], + "seed": 99 if case["case_id"] == "reseeded" else case["seed"], + "status": "succeeded", + "media": {"valid": True}, + "artifact_path": f"artifacts/measurement-r001-c{index:03d}.mp4", + "sha256": _sha(artifacts / f"measurement-r001-c{index:03d}.mp4"), + } + for index, case in enumerate(CASES, 1) + ] + run = {"plan": {**SETTINGS, "cases": CASES}, "records": records} + (bundle / "gpu/c1/baseline/run.json").write_text(json.dumps(run)) + (bundle / "serving-smoke.json").write_text(json.dumps({"cells": [{"concurrency": 1}]})) + (bundle / "manifest.json").write_text(json.dumps({"run_id": "41"})) + (bundle / "SHA256SUMS").write_text( + "".join( + f"{_sha(path)} {path.relative_to(bundle).as_posix()}\n" + for path in sorted(bundle.rglob("*")) + if path.is_file() + ) + ) + shutil.make_archive(str(root / "bundle"), "zip", bundle) + return descriptor, references, bundle + + +def _score(descriptor: Path, references: Path, output: Path, *bundles: str) -> int: + return cli.main( + [ + "score-quality", + *bundles, + "--descriptor", + str(descriptor), + "--references", + str(references), + "--run-id", + "41", + "--source-revision", + REVISION, + "--output", + str(output), + ] + ) + + +@pytest.fixture(scope="module") +def scored(tmp_path_factory): + root = tmp_path_factory.mktemp("quality") + descriptor, references, bundle = _setup(root) + shutil.copytree(bundle, root / "partial") + (root / "partial/manifest.json").unlink() + zipped, directory, partial = f"7={root / 'bundle.zip'}", f"8={bundle}", f"9={root / 'partial'}" + assert _score(descriptor, references, root / "first", zipped, partial, directory) == 0 + assert _score(descriptor, references, root / "second", directory, zipped, partial) == 0 + return root + + +def test_scores_each_measured_clip_against_its_own_reference(scored): + quality = json.loads((scored / "first/quality.json").read_bytes()) + assert (scored / "first/quality.json").read_bytes() == ( + scored / "second/quality.json" + ).read_bytes() + assert ( + f"{_sha(scored / 'first/quality.json')} quality.json" + in (scored / "first/SHA256SUMS").read_text().splitlines() + ) + pairs = { + (p["candidate"]["artifact_id"], p["candidate"]["case_id"]): p for p in quality["pairs"] + } + assert sorted(pairs) == [(7, "changed"), (7, "same"), (8, "changed"), (8, "same")] + for artifact in (7, 8): + same, changed = pairs[artifact, "same"], pairs[artifact, "changed"] + assert same["reference"]["media_path"] == "same.mp4" + assert same["comparison"]["metrics"]["video_identical"] is True + assert changed["candidate"]["media_path"] == ( + "gpu/c1/baseline/artifacts/measurement-r001-c002.mp4" + ) + assert changed["comparison"]["compatible"] is True + assert 0 < changed["comparison"]["metrics"]["video_psnr_db"] < 15 + + +def test_reports_a_clip_generated_with_other_settings_instead_of_scoring_it(scored): + quality = json.loads((scored / "first/quality.json").read_bytes()) + assert [ + ( + item["candidate"]["artifact_id"], + item["candidate"]["case_id"], + item["reason"], + item["fields"], + ) + for item in quality["unscored"] + ] == [ + (7, "reseeded", "settings_mismatch", ["seed"]), + (8, "reseeded", "settings_mismatch", ["seed"]), + ] + + +def test_lists_a_partial_bundle_as_unreadable_and_still_scores_the_others(scored): + quality = json.loads((scored / "first/quality.json").read_bytes()) + assert quality["unreadable"] == [ + {"artifact_id": 9, "error": "FileNotFoundError: manifest.json"} + ] + assert {pair["candidate"]["artifact_id"] for pair in quality["pairs"]} == {7, 8} + + +def test_refuses_a_reference_asset_that_differs_from_the_descriptor(tmp_path): + descriptor, references, bundle = _setup(tmp_path) + tampered = bytearray((references / "changed.mp4").read_bytes()) + tampered[-1] ^= 1 + (references / "changed.mp4").write_bytes(tampered) + assert _score(descriptor, references, tmp_path / "out", f"7={bundle}") == 2 + assert not (tmp_path / "out").exists() From 90bc8427e5529fe287f3c6b86a9f966ce3b3de6c Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Sat, 7 Nov 2026 01:36:00 -0800 Subject: [PATCH 028/111] =?UTF-8?q?[VideoGenX]=20add=20an=20end-to-end=20H?= =?UTF-8?q?3=20submission=20guide=20/=20[VideoGenX]=20=E6=96=B0=E5=A2=9E?= =?UTF-8?q?=20H3=20=E6=8F=90=E4=BA=A4=E5=85=A8=E6=B5=81=E7=A8=8B=E6=8C=87?= =?UTF-8?q?=E5=8D=97=20(#4885)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: add an end-to-end H3 video submission guide * docs: use the standard change and render commands in the video guide * docs: link the render step instead of restating it --- CONTRIBUTING.md | 2 + inferencex-e2e/docs/video-generation.md | 79 +++-------- video-generation/SUBMITTING.md | 172 ++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 60 deletions(-) create mode 100644 video-generation/SUBMITTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a895f04486..5f0f1bb120 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,6 +27,8 @@ Every PR description must include an **AI model disclosure** section inside a co 3. Ping a core maintainer on Slack for final approval, after obtaining the checklist sign-off when required. 4. An authorized maintainer posts `/use ` (see below) and the PR is merged via the reuse path. +H3 video deployments follow the same flow. The [H3 submission guide](video-generation/SUBMITTING.md) lists their video-specific steps in order. + **Performance changelog:** A PR that changes what any benchmark runs **MUST** append one new entry covering the affected config keys to the physical end of `inferencex-e2e/perf-changelog.yaml`; `python -m infx.matrix.changed --base origin/main` (run from `inferencex-e2e/`) lists them. A PR that changes no benchmark point adds none. Historical entries **MUST NOT** be edited. For full config retirement, list the deleted config keys with `retired: true`. The planner verifies that every selected key existed in the base revision and is absent from the current master configs, retains the entry in changelog metadata, and schedules no benchmarks or evals for it. Retirement entries cannot use scenario filters, `append-only`, or eval-selection fields or modifiers; partial scenario retirement uses a regular entry for the surviving config. diff --git a/inferencex-e2e/docs/video-generation.md b/inferencex-e2e/docs/video-generation.md index 604d76b55f..c7f0b2f5d0 100644 --- a/inferencex-e2e/docs/video-generation.md +++ b/inferencex-e2e/docs/video-generation.md @@ -47,6 +47,8 @@ explicitly selected model view, records the final command, then starts it. It adds no engine flags or performance environment defaults. The SGLang child checks the assigned UUID order before importing its engine. The previous B200 `frozen-c124` four-clip JSON and YAML artifacts remain readable as historical evidence. +The App shows their four-sample cells in History, below its ten-valid-sample chart +floor. Each recipe's `readiness.timeout_seconds` sets how long the server may load and warm up. @@ -95,67 +97,24 @@ the first native video reuse request; a PR-local checker cannot bypass that gate ## Contribute a video deployment -For an existing video runtime/API, change its native recipe and master config rather -than adding a launcher or modifying the Python client. The configured deployments -are the `*-video` keys in [`configs/nvidia-master.yaml`](../configs/nvidia-master.yaml) -and [`configs/amd-master.yaml`](../configs/amd-master.yaml); -each names its recipe directory under -[`benchmarks/single_node/srt-slurm-recipes/h3/`](../benchmarks/single_node/srt-slurm-recipes/h3/). -Their presence does not establish final-head GPU qualification. +To add or change a deployment, follow the +[H3 submission guide](../../video-generation/SUBMITTING.md). -1. Choose an existing combination and keep its frozen workload, model revision, - concurrency population and measurement boundary. Put the full engine command - after `services[].args`' `--`; put nonsecret process settings in `services[].env` - or recipe-wide `environment`. Keep the native serve/client entrypoints and - telemetry binding. A new model, engine API, topology or model-view format - needs maintainer adapter validation before it becomes a recipe-only contribution. -2. Change the existing row in its master config, or copy it with a - distinct key and recipe for a deployment that must remain separately selectable. - Pin the image by digest and use its real interpreter and architecture. The - H100/H200/B200 SGLang rows use the public CUDA amd64 image and the MI355X row its - ROCm MI35x build. A shared image does - not qualify a different topology or SKU: validate - the selected deployments through their sweep. Maintainers stage model snapshots in the runner's - existing HF cache; contributors do not patch the cache or SSH extra launch flags. -3. Render locally from `inferencex-e2e/`, using the repository's pinned srt-slurm - submodule and locked dependencies. This writes files only and submits no job: +### Render a deployment locally - ```bash - recipe_output=$(mktemp -d) - uv run --locked --extra recipes infx generate \ - --config-key h3-bf16-h100-sglang-video --output-dir "$recipe_output" - uv run --locked --all-extras --group test python -m infx.matrix.changed \ - --base origin/main --json - ``` +Render locally from `inferencex-e2e/`, using the repository's pinned srt-slurm +submodule and locked dependencies. This writes files only and submits no job: - Inspect `manifest.json` and its generated YAML for the selected config, image, - model, GPU count, full command, environment and client concurrency. The recipe - fingerprint binds the matrix row and rendered recipe; a successful render - proves schema/binding, not model execution. Run the focused binder, entrypoint, - matrix and result tests for changed behavior. -4. Append the PR's single `perf-changelog.yaml` entry with every affected config - key and a `scenario-type` that lists `video-generation`. The planner rejects an - entry whose named video config would otherwise plan no jobs. Follow [the contribution workflow](../../CONTRIBUTING.md) for review, - CODEOWNER sign-off, the primary sweep label and trusted reuse. A maintainer - controls fork scheduling. Use the final PR-selected matrix and preserve failed - attempts; an ad-hoc diagnostic cannot replace its qualifying sweep. -5. Inspect the original run artifacts. `video-sweep-manifest` identifies the - selected attempt and deployments. Each `h3-results-*` bundle retains - `manifest.json`, `ci.json`, `SHA256SUMS` and `gpu/c*/` request, media and power - evidence. A valid `release-v1` deployment needs 20 valid measured C1 clips plus - its separate warmup. Four-sample B200 cells remain History evidence below the - App's ten-valid-sample chart floor. +```bash +recipe_output=$(mktemp -d) +uv run --locked --extra recipes infx generate \ + --config-key h3-bf16-h100-sglang-video --output-dir "$recipe_output" +uv run --locked --all-extras --group test python -m infx.matrix.changed \ + --base origin/main --json +``` -During execution, native `recipe.lock.yaml` and the scheduler job record identify -the submitted configuration. Inspect `logs/video/server/launch.json` for the actual -argv/environment and assigned GPU UUIDs; `spec.json` retains the selected model -and workload. Service startup failures are in `logs/service_video-server.out` and -`logs/sweep_.log`. Client failures remain in each cell's `baseline/run.json`; -native power failures retain `power-validation.json`. Keep run ID, attempt, source -head, executed checkout, recipe fingerprint and plan hash distinct when reporting -a problem. Never hand-edit a bundle to make its result eligible. - -After authorized merge and publication, use the App's published History and -read-only video API to retrieve the original source. An unpublished source is not -an official result, and opening a comparison must not publish it. Follow the -publication handoff above rather than copying media by hand. +Inspect `manifest.json` and its generated YAML for the selected config, image, +model, GPU count, full command, environment and client concurrency. The recipe +fingerprint binds the matrix row and rendered recipe; a successful render +proves schema/binding, not model execution. Run the focused binder, entrypoint, +matrix and result tests for changed behavior. diff --git a/video-generation/SUBMITTING.md b/video-generation/SUBMITTING.md new file mode 100644 index 0000000000..1ae6ebb7b0 --- /dev/null +++ b/video-generation/SUBMITTING.md @@ -0,0 +1,172 @@ +# Submit an H3 video deployment + +Follow these steps to add or change an H3 (VideoGenX) deployment, from choosing a +config to a published result. Each step links the section that holds its detail. +[Native video delivery](../inferencex-e2e/docs/video-generation.md) explains how the +sweep runs, what a result bundle contains, and how reuse is validated. + +| Step | Owner | +| --- | --- | +| [1. Choose the deployment](#1-choose-the-deployment) | You | +| [2. Change the recipe and config row](#2-change-the-recipe-and-config-row) | You | +| [3. Check locally](#3-check-locally) | You | +| [4. Append the changelog entry](#4-append-the-changelog-entry) | You | +| [5. Open the PR and start the sweep](#5-open-the-pr-and-start-the-sweep) | You open the PR. A maintainer labels a fork PR. | +| [6. Confirm the run qualifies](#6-confirm-the-run-qualifies) | You | +| [7. Get review and sign-off](#7-get-review-and-sign-off) | One eligible CODEOWNER, then a core maintainer | +| [8. Reuse and merge](#8-reuse-and-merge) | An authorized maintainer | +| [9. Verify publication](#9-verify-publication) | Merge Ingest publishes. You verify. | + +## 1. Choose the deployment + +The configured deployments are the `*-video` keys in +[`nvidia-master.yaml`](../inferencex-e2e/configs/nvidia-master.yaml) and +[`amd-master.yaml`](../inferencex-e2e/configs/amd-master.yaml). Each key names its +recipe directory under +[`srt-slurm-recipes/h3/`](../inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/). + +Start from an existing combination. Keep its frozen workload, model revision, +concurrency population, and measurement boundary. To change a deployment, edit its +row. To add a deployment that must stay separately selectable, copy the row under a +distinct key with its own recipe. A new model, engine API, topology, or model-view +format needs maintainer adapter validation before it can be a recipe-only +contribution. + +## 2. Change the recipe and config row + +Change the native recipe and the master config row. Do not add a launcher or modify +the Python client. Keep the native serve and client entrypoints and the telemetry +binding. + +- Put the complete engine command after the `--` in `services[].args`. +- Put nonsecret process settings in `services[].env` or the recipe-wide + `environment`. [Native video delivery](../inferencex-e2e/docs/video-generation.md) + lists the placeholders the binder substitutes and the names it rejects. +- Set `services[].env.VIDEO_RUNTIME_REVISION` to the engine's 40-character commit. + The binder rejects any other value, and the server records it as the engine + revision. +- Pin the image by SHA256 digest, and use its real interpreter and architecture. + The H100, H200, and B200 SGLang rows use the public CUDA amd64 image. The MI355X + row uses its ROCm MI35x build. + +A shared image does not qualify a different topology or SKU. Each selected +deployment qualifies through its own sweep. Maintainers stage model snapshots in the +runner's existing Hugging Face cache. Do not patch the cache or pass launch flags +outside the recipe. + +## 3. Check locally + +Render the changed deployments and list the changed config keys as +[Render a deployment locally](../inferencex-e2e/docs/video-generation.md#render-a-deployment-locally) +describes. If you changed code under `video-generation/`, run its +[local checks](README.md#local-checks). For code under `inferencex-e2e/infx/`, run +the [repository local checks](../AGENTS.md#local-checks). + +## 4. Append the changelog entry + +Append one entry to the physical end of +[`perf-changelog.yaml`](../inferencex-e2e/perf-changelog.yaml). List every affected +config key, and include `video-generation` in `scenario-type`. The planner rejects +an entry whose named video configs would otherwise plan no jobs. + +```yaml +- config-keys: + - h3-bf16-h100-sglang-video + scenario-type: [video-generation] + description: + - "" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX +``` + +Leave `append-only` unset, because video runs only as a regular sweep. The +[performance changelog rule](../CONTRIBUTING.md#pr-review-flow) covers the one-entry +limit and byte preservation. + +## 5. Open the PR and start the sweep + +Write the title and body as [PR descriptions](../CONTRIBUTING.md#pr-descriptions) +requires. Then start the sweep. + +- **Same-repository PR.** Apply exactly one primary label, normally + `full-sweep-fail-fast`. `run-sweep.yml` then sweeps the changelog matrix. +- **Fork PR.** Mark the PR ready for review and resolve merge conflicts. A + maintainer applies the label, which approves only the current head and dispatches + the [trusted sweep](../.github/workflows/README.md#trusted-external-fork-sweep-dispatch-poc). + After each push, the maintainer removes and re-adds the label. + +Do not add `evals-only` or `agentx-fast`, because a run with either label cannot be +reused. [PR sweep labels](../.github/workflows/README.md#pr-sweep-labels) describes +every label and modifier. + +## 6. Confirm the run qualifies + +The qualifying run is a full sweep of the final changelog matrix on a commit in the +PR. Its `validate-video-results` job must pass. That job checks the +[reuse conditions](../inferencex-e2e/docs/video-generation.md) against the retained +evidence of every selected deployment. A valid `release-v1` deployment needs 20 +valid measured C1 clips plus its separate warmup. Keep failed attempts. An ad hoc +diagnostic run cannot replace the qualifying sweep. + +Inspect the run's artifacts: + +- `video-sweep-manifest` identifies the selected attempt and deployments. +- Each `h3-results-*` bundle holds `manifest.json`, `ci.json`, `SHA256SUMS`, and the + request, media, and power evidence under `gpu/c*/`. + +Never hand-edit a bundle to make its result eligible. If a deployment fails, see +[Troubleshoot a run](#troubleshoot-a-run). + +## 7. Get review and sign-off + +The master configs and the recipes have vendor owners in +[CODEOWNERS](../.github/CODEOWNERS). One eligible CODEOWNER posts the +[PR Review Checklist](../CONTRIBUTING.md#the-pr-review-checklist-codeowner-sign-off). +A core maintainer gives final approval. + +## 8. Reuse and merge + +After the final sweep passes, an authorized maintainer comments `/use ` to +select it. [Reuse at merge](../CONTRIBUTING.md#reusing-your-prs-green-sweep-at-merge-with-use) +describes the rules. + +- **Same-repository PR.** The maintainer merges with + `infx.workflows.merge_with_reuse`. +- **Fork PR.** The maintainer pins the trusted `e2e-tests.yml` run by its ID, then + merges. Automatic selection and `merge_with_reuse` find only `run-sweep.yml` runs. + +## 9. Verify publication + +On the merge to `main`, Merge Ingest publishes the reused run as +[native video delivery](../inferencex-e2e/docs/video-generation.md) describes. +Confirm that the post-merge jobs pass, as [After merging](../CONTRIBUTING.md#after-merging) +requires. Then use the App's published History and read-only video API to retrieve +the original source. An unpublished source is not an official result, and opening a +comparison must not publish it. Do not copy media by hand. + +## Reference quality (PSNR) + +After a same-repository Run Sweep finishes, the +[Video Quality](../.github/workflows/video-quality.yml) workflow scores each measured +clip against the public Diffusers reference videos. It uploads the scores as the +`h3-quality--` artifact. It does not score a trusted fork sweep +automatically. A maintainer dispatches it with that run's `run_id` and +`run_attempt`. + +PSNR measures how closely each clip matches the reference implementation's output +for the same prompt and seed. Other hardware and engines legitimately diverge from +that output, so the score is not a quality grade. `quality.json` keeps +`quality_status: unjudged` and `release_qualified: false`, and no workflow gates +reuse or merge on the score. +[Reference PSNR](README.md#reference-psnr) describes the method and the reasons a +clip stays unscored. + +## Troubleshoot a run + +During execution, the native `recipe.lock.yaml` and the scheduler job record identify +the submitted configuration. Inspect `logs/video/server/launch.json` for the actual +argv, environment, and assigned GPU UUIDs. `spec.json` retains the selected model +and workload. Service startup failures are in `logs/service_video-server.out` and +`logs/sweep_.log`. Client failures remain in each cell's `baseline/run.json`. +Native power failures retain `power-validation.json`. When you report a problem, +keep the run ID, attempt, source head, executed checkout, recipe fingerprint, and +plan hash distinct. From 5b5ff8ac54d5e472fa76aa18fe291face0e63933 Mon Sep 17 00:00:00 2001 From: Bryan <58582368+Oseltamivir@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:20:25 +0800 Subject: [PATCH 029/111] docs(collectivex): register H100 DeepEP low-latency at EP8 only and record why (#4894) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H100's EFA pool segfaults every rank at EP16 low-latency setup (IBGDA has no EFA transport), so the cell could only ever show as pending. Methodology also records that H100 EP16 normal prefill does not finish inside the per-case guard. 文档：H100 的 DeepEP 低延迟仅注册 EP8，并记录原因。H100 的 EFA 集群在 EP16 低延迟初始化时所有 rank 都会段错误（IBGDA 没有 EFA 传输），该单元只能永远显示为待测。方法学同时记录 H100 EP16 normal prefill 无法在单用例时限内完成。 Co-authored-by: Claude Opus 5.5 --- collectivex/README.md | 4 +- collectivex/configs/platform_config.json | 2 +- collectivex/docs/methodology.md | 5 +- collectivex/results/cases.json | 76 ------------------------ 4 files changed, 7 insertions(+), 80 deletions(-) diff --git a/collectivex/README.md b/collectivex/README.md index c574511f7e..0df5ea930c 100644 --- a/collectivex/README.md +++ b/collectivex/README.md @@ -30,7 +30,7 @@ runs the decode and prefill ladders; `low-latency` runs decode, plus prefill for | Backend | Engine selector (`maturity`) | `normal` | `low-latency` | |---|---|---|---| -| `deepep-v2` | vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` (`production`) | EP8, EP16: H100, H200, B200, B300, GB200, GB300 | EP8, EP16: H100, H200, B200, GB200, GB300 | +| `deepep-v2` | vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` (`production`) | EP8, EP16: H100, H200, B200, B300, GB200, GB300 | EP8: H100. EP8, EP16: H200, B200, GB200, GB300 | | `mori` | vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` (`production`) | EP8, EP16: MI300X, MI325X, MI355X | EP16: MI300X, MI325X, MI355X | | `uccl-ep` | none; engines load it as a drop-in `deep_ep` under vLLM's DeepEP backends (`candidate`) | EP8: H100, H200, B200, MI300X, MI325X, MI355X. EP16: H100, H200, B200, MI300X, MI325X | EP8: B200. EP8, EP16: H100, H200. EP16: MI300X, MI325X | | `nccl-ep` | TensorRT-LLM `NcclEP` (`production` for low-latency rank-major BF16 where it matches, `candidate` otherwise) | EP8, EP16: H100, H200, B200, B300, GB200, GB300 | EP8: B300. EP8, EP16: H100, H200, B200, GB200, GB300 | @@ -39,7 +39,7 @@ runs the decode and prefill ladders; `low-latency` runs decode, plus prefill for Every backend runs a BF16 dispatch control and an FP8 dispatch; nccl-ep's FP8 is its internal DS_FP8E3M4 recipe, low-latency prefill and decode. Combine is always BF16. Coverage is uniform routing only. -`results/cases.json` is the committed copy of the full matrix: 290 requested cases, of which 252 are +`results/cases.json` is the committed copy of the full matrix: 288 requested cases, of which 250 are runnable EP cases, 18 are swap-blocks cases (two per pool), 12 are kv-transfer cases (one per pool, backend and fabric) and 8 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on H100, H200 and B200, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink diff --git a/collectivex/configs/platform_config.json b/collectivex/configs/platform_config.json index b2164b56c9..ad66eb83a8 100644 --- a/collectivex/configs/platform_config.json +++ b/collectivex/configs/platform_config.json @@ -12,7 +12,7 @@ "launcher": "single-slurm", "runner_label": "cluster:h100-dsxe", "backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8, 16], "flashinfer-ep": [8]}, - "ll_backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8, 16]}, + "ll_backends": {"deepep-v2": [8], "uccl-ep": [8, 16], "nccl-ep": [8, 16]}, "fabric": {"nic": "AWS EFA (p5.48xlarge, 32 x 100G per node)", "switch": "AWS EFA fabric (SRD)"}, "operator": { "partition": "batch_1", diff --git a/collectivex/docs/methodology.md b/collectivex/docs/methodology.md index a343acce6f..611d217cb7 100644 --- a/collectivex/docs/methodology.md +++ b/collectivex/docs/methodology.md @@ -223,7 +223,10 @@ and each collective split into a send kernel and a receive hook that runs straig runs over intra-node NVLink; multi-node EP16 rides NVSHMEM IBGDA, which wants gdrdrv on the node (b200-nscale's bare-metal IB has it), and GB EP16 stays inside MNNVL. B300 registers no DeepEP low-latency: the legacy Buffer initializes NVSHMEM with an RDMA transport even on one node, and -NVSHMEM aborts on topology detection over B300's EFA. +NVSHMEM aborts on topology detection over B300's EFA. H100's EFA pool registers EP8 only: at EP16 +every rank segfaults during setup, since IBGDA has no EFA transport. Its normal-mode EP16 prefill +also does not finish inside the per-case guard (`COLLX_RUN_TIMEOUT`) through the EFA proxy, so those +two cells stay unmeasured. ### UCCL-EP diff --git a/collectivex/results/cases.json b/collectivex/results/cases.json index 8527a271c3..1e3574aa76 100644 --- a/collectivex/results/cases.json +++ b/collectivex/results/cases.json @@ -5472,82 +5472,6 @@ "reason": null, "sku": "h100-dgxc" }, - { - "case": { - "backend": "deepep-v2", - "case_id": "h100-dgxc-deepep-v2-deepseek-v4-pro-low-latency-decode-ep16-uniform-bf16", - "ep": 16, - "experts": 384, - "gpus_per_node": 8, - "hidden": 7168, - "ladder": "1 2 4 8 16 32 64 128 256 512", - "mode": "low-latency", - "nodes": 2, - "phase": "decode", - "precision": "bf16", - "routing": "uniform", - "scale_out_transport": "rdma", - "scale_up_domain": 8, - "scale_up_transport": "nvlink", - "scope": "scale-out", - "seed": 67, - "suite": "ep-core", - "timing": { - "chain_drop": 16, - "chain_iters_per_trial": 128, - "chain_trials_per_point": 4, - "iters_per_trial": 8, - "trials_per_point": 256, - "warmup_iters_per_trial": 32 - }, - "topk": 6, - "topology_class": "h100-nvlink-rdma", - "transport": "nvlink-rdma", - "workload": "deepseek-v4-pro" - }, - "detail": null, - "disposition": "runnable", - "reason": null, - "sku": "h100-dgxc" - }, - { - "case": { - "backend": "deepep-v2", - "case_id": "h100-dgxc-deepep-v2-deepseek-v4-pro-low-latency-decode-ep16-uniform-fp8", - "ep": 16, - "experts": 384, - "gpus_per_node": 8, - "hidden": 7168, - "ladder": "1 2 4 8 16 32 64 128 256 512", - "mode": "low-latency", - "nodes": 2, - "phase": "decode", - "precision": "fp8", - "routing": "uniform", - "scale_out_transport": "rdma", - "scale_up_domain": 8, - "scale_up_transport": "nvlink", - "scope": "scale-out", - "seed": 67, - "suite": "ep-core", - "timing": { - "chain_drop": 16, - "chain_iters_per_trial": 128, - "chain_trials_per_point": 4, - "iters_per_trial": 8, - "trials_per_point": 256, - "warmup_iters_per_trial": 32 - }, - "topk": 6, - "topology_class": "h100-nvlink-rdma", - "transport": "nvlink-rdma", - "workload": "deepseek-v4-pro" - }, - "detail": null, - "disposition": "runnable", - "reason": null, - "sku": "h100-dgxc" - }, { "case": { "backend": "uccl-ep", From 7a2db9d088b1db34de27da63fe730db465ba297b Mon Sep 17 00:00:00 2001 From: Bryan <58582368+Oseltamivir@users.noreply.github.com> Date: Sat, 7 Nov 2026 23:14:00 +0800 Subject: [PATCH 030/111] =?UTF-8?q?CollectiveX:=20run=20FlashInfer=20EP8?= =?UTF-8?q?=20on=20B300=20/=20CollectiveX=EF=BC=9A=E5=9C=A8=20B300=20?= =?UTF-8?q?=E4=B8=8A=E8=BF=90=E8=A1=8C=20FlashInfer=20EP8=20(#4895)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(collectivex): run flashinfer-ep EP8 on b300 Hand runs from cx-probe-image-stack on 2026-11-07: BF16 and FP8, decode and prefill, oracle green; within about 1% of b200. 功能：在 b300 上运行 flashinfer-ep EP8。2026-11-07 手动运行，BF16 与 FP8 的 decode 与 prefill 均通过 oracle，与 b200 相差约 1%。 Co-Authored-By: Claude Opus 5.5 * docs(collectivex): add the perf-changelog entry 文档：添加 perf-changelog 条目。 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- collectivex/README.md | 8 +- collectivex/configs/platform_config.json | 2 +- collectivex/docs/methodology.md | 6 +- collectivex/perf-changelog.yaml | 6 + collectivex/results/cases.json | 228 +++++++++++++++++++++++ 5 files changed, 242 insertions(+), 8 deletions(-) diff --git a/collectivex/README.md b/collectivex/README.md index 0df5ea930c..8db085ba04 100644 --- a/collectivex/README.md +++ b/collectivex/README.md @@ -34,15 +34,15 @@ runs the decode and prefill ladders; `low-latency` runs decode, plus prefill for | `mori` | vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` (`production`) | EP8, EP16: MI300X, MI325X, MI355X | EP16: MI300X, MI325X, MI355X | | `uccl-ep` | none; engines load it as a drop-in `deep_ep` under vLLM's DeepEP backends (`candidate`) | EP8: H100, H200, B200, MI300X, MI325X, MI355X. EP16: H100, H200, B200, MI300X, MI325X | EP8: B200. EP8, EP16: H100, H200. EP16: MI300X, MI325X | | `nccl-ep` | TensorRT-LLM `NcclEP` (`production` for low-latency rank-major BF16 where it matches, `candidate` otherwise) | EP8, EP16: H100, H200, B200, B300, GB200, GB300 | EP8: B300. EP8, EP16: H100, H200, B200, GB200, GB300 | -| `flashinfer-ep` | vLLM `--all2all-backend flashinfer_nvlink_one_sided` (`production`) | EP8: H100, H200, B200. EP8, EP16: GB200, GB300 | none | +| `flashinfer-ep` | vLLM `--all2all-backend flashinfer_nvlink_one_sided` (`production`) | EP8: H100, H200, B200, B300. EP8, EP16: GB200, GB300 | none | Every backend runs a BF16 dispatch control and an FP8 dispatch; nccl-ep's FP8 is its internal DS_FP8E3M4 recipe, low-latency prefill and decode. Combine is always BF16. Coverage is uniform routing only. -`results/cases.json` is the committed copy of the full matrix: 288 requested cases, of which 250 are +`results/cases.json` is the committed copy of the full matrix: 294 requested cases, of which 254 are runnable EP cases, 18 are swap-blocks cases (two per pool), 12 are kv-transfer cases (one per pool, -backend and fabric) and 8 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on -H100, H200 and B200, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink +backend and fabric) and 10 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on +H100, H200, B200 and B300, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink domain). A cell is listed as unsupported only when its backend runs on that SKU at another EP size, and low-latency adds only runnable cells, so a missing low-latency cell (DeepEP V2 on B300, for example) diff --git a/collectivex/configs/platform_config.json b/collectivex/configs/platform_config.json index ad66eb83a8..513a90d26d 100644 --- a/collectivex/configs/platform_config.json +++ b/collectivex/configs/platform_config.json @@ -85,7 +85,7 @@ "scale_up_transport": "nvlink", "launcher": "single-slurm", "runner_label": "cluster:b300-dsxe", - "backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16]}, + "backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16], "flashinfer-ep": [8]}, "ll_backends": {"nccl-ep": [8]}, "kv_backends": {"nixl": ["rdma"], "mooncake": ["rdma"]}, "fabric": {"nic": "AWS EFA (p6-b300.48xlarge, 16 x 400G per node)", "switch": "AWS EFA fabric (SRD)"}, diff --git a/collectivex/docs/methodology.md b/collectivex/docs/methodology.md index 611d217cb7..f9c93ade94 100644 --- a/collectivex/docs/methodology.md +++ b/collectivex/docs/methodology.md @@ -32,7 +32,7 @@ library only. Details are under [Backends](#backends). | nccl-ep | production for in-domain zero-copy LL rank-major BF16; library otherwise | graph decode; prefill eager | TensorRT-LLM `NcclEP` with NVIDIA/TensorRT-LLM#20631; HT and FP8 (DS_FP8E3M4) are library rows | | uccl-ep | candidate | normal eager; LL graph | vLLM's DeepEP configuration on UCCL's drop-in `deep_ep`; no engine ships its selector | | mori | production | graph decode; eager prefill | vLLM's MoRI IntraNode / InterNodeV1, LL EP16 only | -| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration; EP8 on H100/H200/B200, EP8 and EP16 on GB200/GB300 | +| flashinfer-ep | production | graph decode; eager prefill | vLLM's integration; EP8 on H100/H200/B200/B300, EP8 and EP16 on GB200/GB300 | ## Timing @@ -307,8 +307,8 @@ point, because two handles from one group share its double-buffer parity slots FlashInfer EP is TensorRT-LLM's one-sided MNNVL `MoeAlltoAll` from FlashInfer 0.7.1, vLLM main's pin: each rank writes tokens directly into its peers' workspace windows, so every rank must sit in one -NVLink domain: EP8 on H100, H200 and B200 (one node), EP8 and EP16 inside MNNVL on GB200/GB300. B300 -is not registered yet. The `flashinfer-mnnvl-…` generation name is kept on the single-node rows. It has one kernel family, so it runs `normal` only, across both ladders. +NVLink domain: EP8 on H100, H200, B200 and B300 (one node), EP8 and EP16 inside MNNVL on +GB200/GB300. The `flashinfer-mnnvl-…` generation name is kept on the single-node rows. It has one kernel family, so it runs `normal` only, across both ladders. It runs vLLM's integration (`flashinfer_nvlink_one_sided`): combine is handed a caller-owned payload with `payload_in_workspace=False`, so FlashInfer's staging copy into the workspace runs, and is timed, inside `combine`, as in vLLM (writing the expert output into the workspace directly is TensorRT-LLM's diff --git a/collectivex/perf-changelog.yaml b/collectivex/perf-changelog.yaml index 12c0ed1521..9f1841646e 100644 --- a/collectivex/perf-changelog.yaml +++ b/collectivex/perf-changelog.yaml @@ -163,3 +163,9 @@ backends: [flashinfer-ep] description: - Run flashinfer-ep EP8 on B200. +- date: 2026-11-07 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4895 + pools: [b300] + backends: [flashinfer-ep] + description: + - Run flashinfer-ep EP8 on B300. diff --git a/collectivex/results/cases.json b/collectivex/results/cases.json index 1e3574aa76..7eec1dfafe 100644 --- a/collectivex/results/cases.json +++ b/collectivex/results/cases.json @@ -1710,6 +1710,82 @@ "reason": null, "sku": "b300" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-decode-ep8-uniform-bf16", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 1, + "phase": "decode", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b300" + }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-decode-ep8-uniform-fp8", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 1, + "phase": "decode", + "precision": "fp8", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b300" + }, { "case": { "backend": "deepep-v2", @@ -1900,6 +1976,82 @@ "reason": null, "sku": "b300" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep8-uniform-bf16", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 1, + "phase": "prefill", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b300" + }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep8-uniform-fp8", + "ep": 8, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 1, + "phase": "prefill", + "precision": "fp8", + "routing": "uniform", + "scale_out_transport": null, + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-up", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-island", + "transport": "nvlink", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "runnable", + "reason": null, + "sku": "b300" + }, { "case": { "backend": "deepep-v2", @@ -2014,6 +2166,44 @@ "reason": null, "sku": "b300" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-decode-ep16-uniform-bf16", + "ep": 16, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1 2 4 8 16 32 64 128 256 512", + "mode": "normal", + "nodes": 2, + "phase": "decode", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": "rdma", + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-out", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-rdma", + "transport": "nvlink-rdma", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "unsupported", + "reason": "backend-platform-unsupported", + "sku": "b300" + }, { "case": { "backend": "deepep-v2", @@ -2128,6 +2318,44 @@ "reason": null, "sku": "b300" }, + { + "case": { + "backend": "flashinfer-ep", + "case_id": "b300-flashinfer-ep-deepseek-v4-pro-normal-prefill-ep16-uniform-bf16", + "ep": 16, + "experts": 384, + "gpus_per_node": 8, + "hidden": 7168, + "ladder": "1024 2048 4096 8192", + "mode": "normal", + "nodes": 2, + "phase": "prefill", + "precision": "bf16", + "routing": "uniform", + "scale_out_transport": "rdma", + "scale_up_domain": 8, + "scale_up_transport": "nvlink", + "scope": "scale-out", + "seed": 67, + "suite": "ep-core", + "timing": { + "chain_drop": 16, + "chain_iters_per_trial": 128, + "chain_trials_per_point": 4, + "iters_per_trial": 8, + "trials_per_point": 256, + "warmup_iters_per_trial": 32 + }, + "topk": 6, + "topology_class": "b300-nvlink-rdma", + "transport": "nvlink-rdma", + "workload": "deepseek-v4-pro" + }, + "detail": null, + "disposition": "unsupported", + "reason": "backend-platform-unsupported", + "sku": "b300" + }, { "case": { "backend": "deepep-v2", From cc38a809474835ff16184b943a415042fa4e543c Mon Sep 17 00:00:00 2001 From: Bryan <58582368+Oseltamivir@users.noreply.github.com> Date: Sun, 8 Nov 2026 00:31:34 +0800 Subject: [PATCH 031/111] fix(collectivex): probe that DeepEP V2 loads the venv build on NCCL >= 2.30.4 (#4896) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups: the vLLM image ships its own deep_ep with ElasticBuffer, so the probe now also requires the loaded module to come from the venv, and checks NCCL >= 2.30.4 as vLLM's installer does. Stale comments corrected; methodology says H100 EP16 prefill stays registered and times out. 修复：探测 DeepEP V2 加载的是 venv 中的构建且 NCCL >= 2.30.4。vLLM 镜像自带含 ElasticBuffer 的 deep_ep，因此探测还要求加载的模块来自 venv，并像 vLLM 安装脚本一样检查 NCCL >= 2.30.4。修正过时 注释；方法学注明 H100 EP16 prefill 仍注册但会超时。 Co-authored-by: Claude Opus 5.5 --- collectivex/docs/methodology.md | 7 ++++--- collectivex/runtime/common.sh | 8 ++++---- collectivex/runtime/prepare_backend.sh | 13 ++++++++++--- 3 files changed, 18 insertions(+), 10 deletions(-) diff --git a/collectivex/docs/methodology.md b/collectivex/docs/methodology.md index f9c93ade94..49145cbe56 100644 --- a/collectivex/docs/methodology.md +++ b/collectivex/docs/methodology.md @@ -194,7 +194,7 @@ and `do_expand=False` into a receive bounded at the next power of two of T, with prefill runs eager, as vLLM runs large prefill batches uncaptured. `combine` copies the batch's rows into the MoE layer's output, as vLLM's finalize does (`library_config.finalize_copy`); rows without that field predate the copy: on b200 EP8 it adds 4-13% to decode `combine`, 4-5% to prefill `combine` -and 1-2% to the prefill pair period (1-5% for uccl-ep). +and 1-2% to the prefill pair period; uccl-ep's pair period moves 1-5%. Scale-up cases request NCCL Device API LSA and fail closed unless the LSA team covers the whole EP world, which is also what makes the hybrid choice match vLLM's autoselect: DeepEP goes hybrid only @@ -225,8 +225,9 @@ runs over intra-node NVLink; multi-node EP16 rides NVSHMEM IBGDA, which wants gd low-latency: the legacy Buffer initializes NVSHMEM with an RDMA transport even on one node, and NVSHMEM aborts on topology detection over B300's EFA. H100's EFA pool registers EP8 only: at EP16 every rank segfaults during setup, since IBGDA has no EFA transport. Its normal-mode EP16 prefill -also does not finish inside the per-case guard (`COLLX_RUN_TIMEOUT`) through the EFA proxy, so those -two cells stay unmeasured. +stays registered (`backend_mode_phases` is per backend, not per pool) but does not finish inside the +per-case guard (`COLLX_RUN_TIMEOUT`) through the EFA proxy: expect those two cases to time out and +stay unmeasured. ### UCCL-EP diff --git a/collectivex/runtime/common.sh b/collectivex/runtime/common.sh index 3e9dcf7d02..f5b013e62a 100644 --- a/collectivex/runtime/common.sh +++ b/collectivex/runtime/common.sh @@ -28,7 +28,7 @@ COLLX_DEEPEP_V2_EFA_COMMIT="a227152ac29513b1c702860745a1d1e0111ae0dc" COLLX_DEEPEP_V2_NCCL_SPEC="image" COLLX_DEEPEP_V2_EFA_NCCL_SPEC="nvidia-nccl-cu13==2.32.3" -# Selects this pool's DeepEP V2 source into COLLX_DEEPEP_V2_REPO/COMMIT/NCCL_SPEC (the venv cache is keyed +# Selects this pool's DeepEP V2 source into COLLX_DEEPEP_V2_REPO/COMMIT/NCCL_SPEC/TORCH_SPEC (the venv cache is keyed # on the commit, so the two sources never share a build). collx_select_deepep_v2_source() { # Opt-in per pool, not per fabric: on h100-dgxc (p5, 32 EFA NICs per node) the fork's @@ -37,6 +37,8 @@ collx_select_deepep_v2_source() { COLLX_DEEPEP_V2_REPO="$COLLX_DEEPEP_V2_EFA_REPO" COLLX_DEEPEP_V2_COMMIT="$COLLX_DEEPEP_V2_EFA_COMMIT" COLLX_DEEPEP_V2_NCCL_SPEC="$COLLX_DEEPEP_V2_EFA_NCCL_SPEC" + # 2.10.0+cu130's bundled CUDA userland poisons the CUDA context during nvshmem symmetric-heap + # init over MNNVL on sm103/driver 580.159.03 (gb300). COLLX_DEEPEP_V2_TORCH_SPEC="torch==2.11.0" } @@ -45,9 +47,7 @@ collx_select_deepep_v2_source() { # (gb300); every CUDA call after buffer creation fails cudaErrorUnknown. Part of the venv cache key. COLLX_DEEPEP_V2_NVSHMEM_SPEC="nvidia-nvshmem-cu13==3.4.5" -# "image" builds against the image's torch (vLLM v0.32.0: 2.13.0). The fork's venv pins its own: -# 2.10.0+cu130's bundled CUDA userland poisons the CUDA context during nvshmem symmetric-heap -# init over MNNVL on sm103/driver 580.159.03 (gb300). Part of the venv cache key. +# "image" builds against the image's torch (vLLM v0.32.0: 2.13.0). Part of the venv cache key. COLLX_DEEPEP_V2_TORCH_SPEC="image" # Build-recipe generation for the DeepEP venv cache key: bump when build flags change without diff --git a/collectivex/runtime/prepare_backend.sh b/collectivex/runtime/prepare_backend.sh index ab5391b042..0f26d0d828 100644 --- a/collectivex/runtime/prepare_backend.sh +++ b/collectivex/runtime/prepare_backend.sh @@ -191,8 +191,8 @@ deepep_cache_root() { deepep_ready() { [ -f "$1/.ready" ] && [ -x "$1/venv/bin/python" ] && [ -d "$1/source" ]; } # Runtime images (vLLM's) ship a trimmed CUDA toolkit without the cuSPARSE and similar headers that -# torch's ATen/cuda headers include, which the extension build needs. The venv's torch already -# pulled the matching CUDA wheels, so link only the headers the toolkit lacks into +# torch's ATen/cuda headers include, which the extension build needs. The torch the venv sees +# (its own, or the image's) carries the matching CUDA wheels, so link only the headers the toolkit lacks into # /cuda-include: nothing shadows the toolkit's own headers, and full-toolkit images leave # the directory empty. deepep_cuda_header_overlay() { @@ -262,8 +262,15 @@ deepep_activate() { deepep_probe() { "$VIRTUAL_ENV/bin/python" - <<'PY' import inspect +import os import deep_ep +from importlib import metadata assert inspect.isclass(deep_ep.ElasticBuffer) +# The vLLM image also ships a deep_ep with ElasticBuffer; the venv's build must be the one loaded. +assert os.path.realpath(deep_ep.__file__).startswith(os.path.realpath(os.environ["VIRTUAL_ENV"]) + os.sep), deep_ep.__file__ +# ElasticBuffer needs NCCL >= 2.30.4 at build and run time, whichever site-packages supplies it. +nccl = tuple(int(part) for part in metadata.version("nvidia-nccl-cu13").split(".")[:3]) +assert nccl >= (2, 30, 4), nccl PY } @@ -290,7 +297,7 @@ deepep_install() { --extra-index-url https://pypi.org/simple "$COLLX_DEEPEP_V2_TORCH_SPEC" >&2 2>&1 \ || { collx_log "ERROR: $COLLX_DEEPEP_V2_TORCH_SPEC (cu130) installation failed"; return 1; } fi - # Torch pins an older NCCL than ElasticBuffer needs; the vLLM image already carries >= 2.30.4. + # Only the EFA fork installs its own NCCL (2.32.3); "image" uses the image's, checked by deepep_probe. if [ "$COLLX_DEEPEP_V2_NCCL_SPEC" != image ]; then "${pip[@]}" --force-reinstall --no-deps "$COLLX_DEEPEP_V2_NCCL_SPEC" >&2 2>&1 \ || { collx_log "ERROR: $COLLX_DEEPEP_V2_NCCL_SPEC installation failed"; return 1; } From e54148a3d1b2557c253f3e950852d2a77244bf61 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 11:57:18 -0500 Subject: [PATCH 032/111] =?UTF-8?q?[TileRT]=20DeepSeek-V4.1-Flash=20FP4=20?= =?UTF-8?q?MI355X=20AgentX:=20vLLM=20prefill=20+=20TileRT=20decode=20/=20[?= =?UTF-8?q?TileRT]=20DeepSeek-V4.1-Flash=20FP4=20MI355X=20AgentX=EF=BC=9Av?= =?UTF-8?q?LLM=20prefill=20+=20TileRT=20decode=20(#4756)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../dsv41flash-tilert-chat-template.jinja | 286 ++++++++++++++++++ .../agentx/disagg-1p1d-tp4-tp8-mtp.yaml | 124 ++++++++ inferencex-e2e/configs/amd-master.yaml | 31 ++ .../infx/srt_slurm/spec_identity.py | 4 +- .../infx/srt_slurm/synthetic_acceptance.py | 3 +- .../test_spec_identity.py | 7 +- .../srt_slurm/test_synthetic_acceptance.py | 15 + inferencex-e2e/perf-changelog.yaml | 11 + 8 files changed, 478 insertions(+), 3 deletions(-) create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-tilert-chat-template.jinja create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/tilert/mi355x-fp4/agentx/disagg-1p1d-tp4-tp8-mtp.yaml diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-tilert-chat-template.jinja b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-tilert-chat-template.jinja new file mode 100644 index 0000000000..06332f53cc --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-tilert-chat-template.jinja @@ -0,0 +1,286 @@ +{#- Based on chat_template.jinja of the model repo deepseek-ai/DeepSeek-V4.1-Flash (revision + 314113f8a7dadb388215396ba5fbee2c610a04d6). Changed: the 8 lines after this notice are added; + everything below them is that file, unchanged. Its license: + + MIT License + + Copyright (c) 2023 DeepSeek + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. +-#} +{#- The model repo's chat_template.jinja with two changes, so that it renders what vLLM's Python + DeepSeek-V4.1 tokenizer mode renders: the generation prompt is always added (vLLM ignores + add_generation_prompt), and `thinking` is read as vLLM reads it (thinking on when `thinking` or + `enable_thinking` is true; when either key is given and neither is true, chat mode). -#} +{%- set add_generation_prompt = true -%} +{%- if thinking_mode is not defined and thinking is defined -%} + {%- set thinking_mode = "thinking" if (thinking or (enable_thinking is defined and enable_thinking)) else "chat" -%} +{%- endif -%} +{#- DeepSeek-V4.1 chat template, ported from encoding/encoding.py in the model repo. + Extra kwargs: thinking_mode ("thinking" | "chat", default "thinking"; enable_thinking=true/false is accepted + as an alias), drop_thinking (default true), reasoning_effort (int in [1, 100] or "low"/"medium"/"high"/"max", + default "high"). Assistant reasoning is read from reasoning_content, or from thinking as an alias. -#} +{%- set IMG = "<｜deepseek_image｜>" -%} +{%- set DSML = "｜DSML｜" -%} + +{%- if thinking_mode is not defined -%} + {%- set thinking_mode = "chat" if (enable_thinking is defined and enable_thinking is not none and not enable_thinking) else "thinking" -%} +{%- endif -%} +{%- if thinking_mode not in ["chat", "thinking"] -%} + {{- raise_exception("Invalid thinking_mode `" ~ thinking_mode ~ "`, should be \"chat\" or \"thinking\"") -}} +{%- endif -%} +{%- if drop_thinking is not defined -%}{%- set drop_thinking = true -%}{%- endif -%} +{%- if add_generation_prompt is not defined -%}{%- set add_generation_prompt = false -%}{%- endif -%} +{%- if tools is not defined -%}{%- set tools = none -%}{%- endif -%} + +{%- if reasoning_effort is not defined or reasoning_effort is none -%}{%- set reasoning_effort = "high" -%}{%- endif -%} +{%- set effort_map = {"low": 50, "medium": 62, "high": 75, "max": 100} -%} +{%- if reasoning_effort is string and reasoning_effort in effort_map -%} + {%- set budget = effort_map[reasoning_effort] -%} +{%- elif reasoning_effort is integer and 1 <= reasoning_effort <= 100 -%} + {%- set budget = reasoning_effort -%} +{%- else -%} + {{- raise_exception("Invalid reasoning effort: " ~ reasoning_effort ~ ", should be int within [1,100] or one of \"low\", \"medium\", \"high\", \"max\"") -}} +{%- endif -%} + +{#- Text of text/image blocks joined by blank lines; other block types are dropped. -#} +{%- macro joined_text(blocks) -%} + {%- set jt = namespace(parts=[]) -%} + {%- for b in blocks -%} + {%- if b.type == "text" -%} + {%- if IMG in (b.text or "") -%}{{- raise_exception("Text block contains image placeholder " ~ IMG ~ ". Images should be separate content blocks.") -}}{%- endif -%} + {%- set jt.parts = jt.parts + [b.text or ""] -%} + {%- elif b.type in ["image", "image_url"] -%} + {%- set jt.parts = jt.parts + [IMG] -%} + {%- endif -%} + {%- endfor -%} + {{- jt.parts | join("\n\n") -}} +{%- endmacro -%} + +{#- Splits/validates "ns::name" against an explicit namespace and returns the qualified name. -#} +{%- macro qualified_name(name, ns) -%} + {%- if ns is mapping -%}{%- set ns = ns.name -%}{%- endif -%} + {%- set parts = name.split("::", 1) -%} + {%- if parts | length == 2 -%} + {%- if ns is not none and ns != parts[0] -%}{{- raise_exception("Conflicting tool namespaces: " ~ ns ~ " != " ~ parts[0]) -}}{%- endif -%} + {%- set ns = parts[0] -%} + {%- set name = parts[1] -%} + {%- endif -%} + {%- if "::" in name -%}{{- raise_exception("Tool name must not contain '::': " ~ name) -}}{%- endif -%} + {%- if ns is not none and "::" in ns -%}{{- raise_exception("Tool namespace must not contain '::': " ~ ns) -}}{%- endif -%} + {{- name if ns is none else ns ~ "::" ~ name -}} +{%- endmacro -%} + +{%- macro render_tools(tool_list) -%} + {%- set ts = namespace(schemas=[]) -%} + {%- for tool in tool_list -%} + {%- set fn = tool.function if tool.function is defined else tool -%} + {%- set ns = tool.namespace if tool.namespace is defined and tool.namespace is not none else fn.get("namespace") -%} + {%- set schema = namespace(d={}) -%} + {%- for k, v in fn.items() if k != "namespace" -%}{%- set schema.d = dict(schema.d, **{k: v}) -%}{%- endfor -%} + {%- set schema.d = dict(schema.d, name=qualified_name(fn.name, ns)) -%} + {%- if ns is mapping and ns.description -%} + {%- set schema.d = dict(schema.d, description=ns.description ~ "\n" ~ (fn.description or "")) -%} + {%- endif -%} + {%- set ts.schemas = ts.schemas + [schema.d | tojson] -%} + {%- endfor -%} + {{- "## Tools\n\nYou have access to a set of tools to help answer the user's question. You can invoke tools by writing a \"<｜DSML｜ calls>\" block like the following:\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"$TOOL_NAME\">\n<｜DSML｜ parameter name=\"$PARAMETER_NAME\" string=\"true|false\">$PARAMETER_VALUE\n...\n\n<｜DSML｜ invoke name=\"$TOOL_NAME2\">\n...\n\n\n\nString parameters should be specified as is and set `string=\"true\"`. For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string=\"false\"`.\n\nIf thinking_mode is enabled (triggered by ), you MUST output your complete reasoning inside ... BEFORE any tool calls or final response.\n\nOtherwise, output directly after with tool calls or final response.\n\n### Available Tool Schemas\n\n" -}} + {{- ts.schemas | join("\n") -}} + {{- "\n\nYou MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls.\n" -}} +{%- endmacro -%} + +{#- 1. Merge tool results (and consecutive user turns) into user messages made of content blocks. -#} +{%- set mns = namespace(list=[]) -%} +{%- for msg in messages -%} + {%- set content = msg.get("content") -%} + {%- set is_blocks = content is iterable and content is not string and content is not mapping -%} + {%- if content is string and IMG in content -%} + {{- raise_exception("Message content contains image special token " ~ IMG ~ ". Images should be provided as image content blocks.") -}} + {%- endif -%} + {%- set rc = msg.reasoning_content if msg.get("reasoning_content") is not none else msg.get("thinking") -%} + {%- if rc is string and IMG in rc -%} + {{- raise_exception("reasoning_content contains image special token " ~ IMG) -}} + {%- endif -%} + {%- set prev = mns.list[-1] if mns.list else none -%} + {%- if msg.role == "tool" -%} + {%- set tblock = {"type": "tool_result", "tool_use_id": msg.get("tool_call_id", ""), "content": joined_text(content) if is_blocks else msg.get("content", "")} -%} + {%- if prev is not none and prev.role == "user" -%} + {%- set mns.list = mns.list[:-1] + [dict(prev, content_blocks=prev.content_blocks + [tblock])] -%} + {%- else -%} + {%- set mns.list = mns.list + [{"role": "user", "content_blocks": [tblock]}] -%} + {%- endif -%} + {%- elif msg.role == "user" -%} + {%- set blocks = content if is_blocks else [{"type": "text", "text": content or ""}] -%} + {%- if prev is not none and prev.role == "user" and prev.get("task") is none -%} + {%- set mns.list = mns.list[:-1] + [dict(prev, content_blocks=prev.content_blocks + blocks)] -%} + {%- else -%} + {%- set mns.list = mns.list + [dict(msg, content_blocks=blocks)] -%} + {%- endif -%} + {%- else -%} + {%- set mns.list = mns.list + [msg] -%} + {%- endif -%} +{%- endfor -%} + +{#- Top-level tools attach to a leading system message, or to a new empty one. -#} +{%- if tools -%} + {%- if mns.list and mns.list[0].role == "system" -%} + {%- if not mns.list[0].get("tools") -%}{%- set mns.list = [dict(mns.list[0], tools=tools)] + mns.list[1:] -%}{%- endif -%} + {%- else -%} + {%- set mns.list = [{"role": "system", "content": "", "tools": tools}] + mns.list -%} + {%- endif -%} +{%- endif -%} + +{#- 2. Reorder tool results to match the tool_calls order of the preceding assistant message. -#} +{%- set sns = namespace(list=[], ids=[]) -%} +{%- for m in mns.list -%} + {%- if m.role == "assistant" and m.get("tool_calls") -%} + {%- set sns.ids = [] -%} + {%- for tc in m.tool_calls -%} + {%- set sns.ids = sns.ids + [tc.get("id") or tc.get("function", {}).get("id", "")] -%} + {%- endfor -%} + {%- set sns.list = sns.list + [m] -%} + {%- elif m.role == "user" and m.content_blocks -%} + {%- set tb = namespace(blocks=[]) -%} + {%- for b in m.content_blocks if b.type == "tool_result" -%} + {%- set order = namespace(idx=0) -%} + {%- for tc_id in sns.ids -%} + {%- if tc_id and tc_id == b.get("tool_use_id", "") -%}{%- set order.idx = loop.index0 -%}{%- endif -%} + {%- endfor -%} + {%- set tb.blocks = tb.blocks + [dict(b, _order=order.idx)] -%} + {%- endfor -%} + {%- if tb.blocks | length > 1 -%} + {%- set sorted_blocks = tb.blocks | sort(attribute="_order") -%} + {%- set rb = namespace(blocks=[], i=0) -%} + {%- for b in m.content_blocks -%} + {%- if b.type == "tool_result" -%} + {%- set rb.blocks = rb.blocks + [sorted_blocks[rb.i]] -%} + {%- set rb.i = rb.i + 1 -%} + {%- else -%} + {%- set rb.blocks = rb.blocks + [b] -%} + {%- endif -%} + {%- endfor -%} + {%- set sns.list = sns.list + [dict(m, content_blocks=rb.blocks)] -%} + {%- else -%} + {%- set sns.list = sns.list + [m] -%} + {%- endif -%} + {%- else -%} + {%- set sns.list = sns.list + [m] -%} + {%- endif -%} +{%- endfor -%} +{%- set msgs = sns.list -%} + +{#- Mid-conversation system messages count as user turns. Tools anywhere disable drop_thinking. -#} +{%- set state = namespace(last_user=-1, has_tools=false) -%} +{%- for m in msgs -%} + {%- if m.role == "user" or (m.role == "system" and loop.index0 > 0) -%}{%- set state.last_user = loop.index0 -%}{%- endif -%} + {%- if m.get("tools") -%}{%- set state.has_tools = true -%}{%- endif -%} +{%- endfor -%} +{%- set effective_drop = drop_thinking and not state.has_tools -%} +{%- set thinking = thinking_mode == "thinking" -%} + +{{- "<｜begin▁of▁sentence｜>" -}} +{#- 3. Render each message. -#} +{%- for m in msgs -%} + {%- set idx = loop.index0 -%} + {%- set content = m.get("content") -%} + {%- if content is iterable and content is not string and content is not mapping -%}{%- set content = joined_text(content) -%}{%- endif -%} + + {%- if idx == 0 -%} + {%- if thinking or m.role == "system" -%}{{- "<｜System｜>" -}}{%- endif -%} + {%- if thinking -%}{{- "Reasoning Effort: " ~ budget ~ " (range 1-100, the higher the value, the more thorough the reasoning)\n\n" -}}{%- endif -%} + {%- endif -%} + + {%- if m.role == "system" -%} + {%- if idx > 0 -%}{{- "<｜System｜>" -}}{%- endif -%} + {{- content or "" -}} + {%- if m.get("tools") -%}{{- "\n\n" ~ render_tools(m.tools) -}}{%- endif -%} + {%- if m.get("response_format") -%} + {{- "\n\n## Response Format:\n\nYou MUST strictly adhere to the following schema to reply:\n" ~ (m.response_format | tojson) -}} + {%- endif -%} + + {%- elif m.role == "user" -%} + {{- "<｜User｜>" -}} + {%- set up = namespace(parts=[]) -%} + {%- for b in m.content_blocks -%} + {%- if b.type == "text" -%} + {%- if IMG in (b.text or "") -%}{{- raise_exception("Text block contains image placeholder " ~ IMG ~ ". Images should be separate content blocks.") -}}{%- endif -%} + {%- set up.parts = up.parts + [b.text or ""] -%} + {%- elif b.type in ["image", "image_url"] -%} + {%- set up.parts = up.parts + [IMG] -%} + {%- elif b.type == "tool_result" -%} + {%- set tc = b.get("content", "") -%} + {%- if tc is iterable and tc is not string and tc is not mapping -%} + {%- set sub = namespace(parts=[]) -%} + {%- for s in tc -%} + {%- if s.type == "text" -%}{%- set sub.parts = sub.parts + [s.get("text", "")] -%} + {%- elif s.type in ["image", "image_url"] -%}{%- set sub.parts = sub.parts + [IMG] -%} + {%- else -%}{%- set sub.parts = sub.parts + ["[Unsupported " ~ s.type ~ "]"] -%} + {%- endif -%} + {%- endfor -%} + {%- set tc = sub.parts | join("\n\n") -%} + {%- endif -%} + {%- set up.parts = up.parts + ["" ~ tc ~ ""] -%} + {%- else -%} + {%- set up.parts = up.parts + ["[Unsupported " ~ b.type ~ "]"] -%} + {%- endif -%} + {%- endfor -%} + {{- up.parts | join("\n\n") -}} + + {%- elif m.role == "latest_reminder" -%} + {{- "<｜latest_reminder｜>" ~ content -}} + + {%- elif m.role == "assistant" -%} + {%- set prev_has_task = idx > 0 and msgs[idx - 1].get("task") is not none -%} + {%- if thinking and not prev_has_task and (not effective_drop or idx > state.last_user) -%} + {{- ((m.reasoning_content if m.get("reasoning_content") is not none else m.get("thinking")) or "") ~ "" -}} + {%- endif -%} + {{- content or "" -}} + {%- if m.get("tool_calls") -%} + {%- set tcs = namespace(lines=[]) -%} + {%- for tc in m.tool_calls -%} + {%- set fn = tc.function -%} + {%- set params = namespace(lines=[]) -%} + {%- set args = fn.arguments if fn.arguments is mapping else {"arguments": fn.arguments} -%} + {%- for k, v in args.items() -%} + {%- set params.lines = params.lines + ["<" ~ DSML ~ " parameter name=\"" ~ k ~ "\" string=\"" ~ ("true" if v is string else "false") ~ "\">" ~ (v if v is string else v | tojson) ~ ""] -%} + {%- endfor -%} + {%- set name = qualified_name(fn.name, tc.get("namespace") or fn.get("namespace")) -%} + {%- set tcs.lines = tcs.lines + ["<" ~ DSML ~ " invoke name=\"" ~ name ~ "\">\n" ~ params.lines | join("\n") ~ "\n"] -%} + {%- endfor -%} + {{- "\n\n<" ~ DSML ~ " calls>\n" ~ tcs.lines | join("\n") ~ "\n" -}} + {%- endif -%} + {%- if not m.get("wo_eos", false) -%}{{- "<｜end▁of▁sentence｜>" -}}{%- endif -%} + + {%- else -%} + {{- raise_exception("Unknown role: " ~ m.role) -}} + {%- endif -%} + + {#- Transition tokens, only when the next message is an assistant/latest_reminder turn or this is the last one. -#} + {%- if loop.last or msgs[idx + 1].role in ["assistant", "latest_reminder"] -%} + {%- set task = m.get("task") -%} + {%- if task is not none -%} + {%- set task_tokens = {"action": "<｜action｜>", "query": "<｜query｜>", "authority": "<｜authority｜>", "domain": "<｜domain｜>", "title": "<｜title｜>", "read_url": "<｜read_url｜>"} -%} + {%- if task not in task_tokens -%}{{- raise_exception("Invalid task: '" ~ task ~ "'") -}}{%- endif -%} + {%- if task == "action" -%}{{- "<｜Assistant｜>" ~ ("" if thinking else "") -}}{%- endif -%} + {{- task_tokens[task] -}} + {%- elif (m.role == "user" or (m.role == "system" and idx > 0)) and (add_generation_prompt or not loop.last) -%} + {{- "<｜Assistant｜>" ~ ("" if thinking and (not effective_drop or idx >= state.last_user) else "") -}} + {%- endif -%} + {%- endif -%} +{%- endfor -%} diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/tilert/mi355x-fp4/agentx/disagg-1p1d-tp4-tp8-mtp.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/tilert/mi355x-fp4/agentx/disagg-1p1d-tp4-tp8-mtp.yaml new file mode 100644 index 0000000000..a74a423863 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/tilert/mi355x-fp4/agentx/disagg-1p1d-tp4-tp8-mtp.yaml @@ -0,0 +1,124 @@ +schema: 2 +name: dsv41flash-mi355x-tilert-agentx +slurm: + time_limit: "08:00:00" +resources: + gpu_type: mi355x + gpus_per_node: 8 +setup_script: glm5.3-tilert-rocm.sh +environment: + RDMAV_FORK_SAFE: "1" + NCCL_SOCKET_IFNAME: "@fabric.socket-ifname" + GLOO_SOCKET_IFNAME: "@fabric.socket-ifname" + NCCL_IB_HCA: "@fabric.rdma-devices" +roles: + prefill: + engine: + type: vllm + set_visible_devices: true + # The vLLM nightly's PyPI Mooncake sends cross-node writes over HIP IPC and its ionic + # provider lacks kernel ABI 4; this image carries a multi-protocol build and a newer provider. + container: ghcr.io/tile-ai/tilert-rocm-prefill:0.1.7@sha256:8a88e2aa51105facd103217b0dc353258f3fc9368318c3bb62f0400bb8dd1ed5 + nodes: 1 + workers: 1 + gpus: 4 + env: + TILERT_ROLE: prefill + VLLM_ROCM_USE_AITER: "1" + VLLM_ROCM_USE_AITER_MOE: "1" + VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE + AITER_TRITON_LOG_LEVEL: ERROR + VLLM_USE_BREAKABLE_CUDAGRAPH: "1" + OMP_NUM_THREADS: "1" + PYTHONUNBUFFERED: "1" + VLLM_ENGINE_READY_TIMEOUT_S: "3600" + # Register the VRAM staging buffer with the ionic HCAs through dmabuf. + MOONCAKE_DISABLE_HIP_DMABUF: "0" + # vLLM's Rust front end (as the MI355X vLLM DeepSeek-V4.1-Flash recipes run it): parses the + # router's token-id request and returns the first token without the Python front end's + # detokenization of the whole prompt. + VLLM_USE_RUST_FRONTEND: "1" + args: + served-model-name: deepseek-ai/DeepSeek-V4.1-Flash + tensor-parallel-size: 4 + language-model-only: true + tokenizer-mode: deepseek_v41 + # The router reads the first token id from the prefill's logprobs; it asks for token ids per + # request (return_tokens_as_token_ids), which the Rust front end supports. + moe-backend: aiter + gpu-memory-utilization: 0.85 + attention-config: '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' + block-size: 128 + # The DSpark draft runs on the prefill too, so the decode receives its window state. + speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}' + max-model-len: 1048576 + max-num-batched-tokens: 32768 + max-num-seqs: 1 + engram-config: '{"cpu_offload":false}' + enable-prefix-caching: true + # CUDA graphs cover the prefill's step sizes too: an AgentX turn computes a few thousand new + # tokens, and in eager mode such a step is bound by launching its ~2.5K kernels. + compilation-config: '{"cudagraph_capture_sizes":[1,2,4,8,16,32,64,128,256,512,768,1024,1536,2048,3072,4096,6144,8192]}' + kv-transfer-config: >- + {"kv_connector":"DsV41TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.dsv41_prefill_connector", + "kv_role":"kv_producer","kv_connector_extra_config":{ + "tilert_model":"dsv41","tilert_max_seq_len":1048576,"tilert_transport":"mooncake"}} + decode: + engine: tilert + container: ghcr.io/tile-ai/tilert-rocm-decode:0.1.6@sha256:48c30d7636a561b4edf781ac3c7712e09fa9a0b2f1292863b72c2d8e4bcb492a + nodes: 1 + workers: 1 + gpus: 8 + env: + TILERT_ROLE: decode + # The dsv41 profile lands the whole prompt KV in one receive buffer. + TILERT_PD_SHARDS: "1" + # Wake the stream on every token instead of the 200 ms default, so a short + # request's tokens are not held until it ends. + TILERT_DECODE_POLL_MS: "2" + args: + model: dsv41 + # TileRT reads the Hugging Face checkpoint directly; no converted weights. The cluster + # stages no DeepSeek-V4.1-Flash checkpoint, so the job is in Hub mode and mounts nothing + # at /model: read the shared Hub cache's snapshot instead (refs/main, the revision the + # prefill's vLLM loads and the chat template is based on). + model-weights-dir: /hf_hub_cache/hub/models--deepseek-ai--DeepSeek-V4.1-Flash/snapshots/314113f8a7dadb388215396ba5fbee2c610a04d6 + max-seq-len: 1048576 + transport: mooncake + with-mtp: true + num-mtp: 5 +frontend: + type: tilert-router + container_image: ghcr.io/tile-ai/tilert-rocm-decode:0.1.6@sha256:48c30d7636a561b4edf781ac3c7712e09fa9a0b2f1292863b72c2d8e4bcb492a + enable_multiple_frontends: false + env: + TILERT_ROLE: router + args: + parser: none + model-path: /hf_hub_cache/hub/models--deepseek-ai--DeepSeek-V4.1-Flash/snapshots/314113f8a7dadb388215396ba5fbee2c610a04d6 + queue-timeout: 1800 + # The router renders the chat template and tokenizes each request itself, so the prefill + # receives token ids. The template renders what vLLM's deepseek_v41 tokenizer mode renders; the + # router checks that against the prefill's /tokenize at startup and stays off otherwise. + chat-template: /configs/dsv41flash-tilert-chat-template.jinja +sbatch_directives: + cpus-per-task: "128" + mem: "0" +srun_options: + mem: "0" + container-writable: "" + container-remap-root: "" +health_check: + max_attempts: 2160 + interval_seconds: 5 +benchmark: + env: + TOKENIZER: /hf_hub_cache/hub/models--deepseek-ai--DeepSeek-V4.1-Flash/snapshots/314113f8a7dadb388215396ba5fbee2c610a04d6 + SERVED_MODEL_NAME: deepseek-ai/DeepSeek-V4.1-Flash + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" + # Thinking on with high effort, the regime of the golden acceptance curve. + AIPERF_EXTRA_INPUTS: '{"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}' + HF_HOME: /logs/hf_cache + HF_HUB_OFFLINE: "" + TRANSFORMERS_VERBOSITY: error + TOKENIZERS_PARALLELISM: "false" diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 02c05727c3..17ed3e0570 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -1020,6 +1020,37 @@ glm5.3-fp8-mi355x-tilert-agentic: ep: 1 dp-attn: false +dsv41flash-fp4-mi355x-tilert-agentic-dspark: + image: ghcr.io/tile-ai/tilert-rocm-decode:0.1.6@sha256:48c30d7636a561b4edf781ac3c7712e09fa9a0b2f1292863b72c2d8e4bcb492a + model: deepseek-ai/DeepSeek-V4.1-Flash + model-prefix: dsv41flash + runner: cluster:mi355x-amds + precision: fp4 + framework: tilert + router: { name: tilert-pd-router, version: "0.1.7.post2" } + multinode: true + disagg: true + kv-p2p-transfer: mooncake + srt-recipe-dir: dsv41flash/tilert/mi355x-fp4/agentx + scenarios: + agentic-coding: + - search-space: + - spec-decoding: "mtp" + conc-list: [1] + srt-recipe: disagg-1p1d-tp4-tp8-mtp.yaml + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_IMAGE=ghcr.io/tile-ai/tilert-rocm-prefill:0.1.7@sha256:8a88e2aa51105facd103217b0dc353258f3fc9368318c3bb62f0400bb8dd1ed5" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + # SGLang cookbook MI355X FP8 Low Latency recipe on four GPUs: HBM-only KV up to # c32, the cookbook HiCache L1 + L2 tier above it. glm5.3flash-fp8-mi355x-sglang-agentic-mtp: diff --git a/inferencex-e2e/infx/srt_slurm/spec_identity.py b/inferencex-e2e/infx/srt_slurm/spec_identity.py index 4b22d67f13..11090741b5 100644 --- a/inferencex-e2e/infx/srt_slurm/spec_identity.py +++ b/inferencex-e2e/infx/srt_slurm/spec_identity.py @@ -186,7 +186,9 @@ def audit( def spec_arguments(role: Mapping[str, Any]) -> dict[str, Any]: """The speculative arguments of a role, as the snapshot records them.""" - return {k: v for k, v in (role.get("args") or {}).items() if k.startswith(SPEC_ARGUMENTS)} + args = (role.get("args") or {}).items() + # TileRT's draft method follows its model profile. + return {k: v for k, v in args if k.startswith(SPEC_ARGUMENTS) or k == "model"} def snapshot(matrix: list[Mapping[str, Any]], root: Path) -> list[dict[str, Any]]: diff --git a/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py b/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py index 4373b27f42..75c3bf1f2c 100644 --- a/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py +++ b/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py @@ -71,7 +71,8 @@ def spec_parameters(role: Mapping[str, Any], engine: str) -> dict[str, Any]: if engine == "tilert": if args.get("with-mtp") is not True: return {} - return {"method": "mtp", "num_speculative_tokens": args.get("num-mtp")} + method = "dspark" if args.get("model") == "dsv41" else "mtp" + return {"method": method, "num_speculative_tokens": args.get("num-mtp")} if engine == "atom": method = args.get("method") if not method: diff --git a/inferencex-e2e/infx/tests/golden_al_distribution/test_spec_identity.py b/inferencex-e2e/infx/tests/golden_al_distribution/test_spec_identity.py index 67f0c8e767..ab6524f3d8 100644 --- a/inferencex-e2e/infx/tests/golden_al_distribution/test_spec_identity.py +++ b/inferencex-e2e/infx/tests/golden_al_distribution/test_spec_identity.py @@ -10,7 +10,7 @@ load_identities, parse_identity, ) -from infx.srt_slurm.spec_identity import audit, run_collection +from infx.srt_slurm.spec_identity import audit, run_collection, spec_arguments DEFAULT = {"value": None, "source": "default"} CHAIN = {"value": "chain", "source": "documented"} @@ -206,3 +206,8 @@ def test_precision_and_kv_cache_dtypes_are_read_per_engine() -> None: config = {"decoding_type": "MTP", "max_draft_len": 3} trt = _role(speculative_config=config, kv_cache_config={"dtype": "fp8"}) assert run_collection("dsv4", trt, "dynamo-trt").kv_cache_dtype == Fact("fp8", "explicit") + + +def test_snapshot_arguments_keep_the_tilert_model_profile() -> None: + role = _role(model="dsv41", **{"with-mtp": True, "num-mtp": 5, "model-weights-dir": "/w"}) + assert spec_arguments(role) == {"model": "dsv41", "with-mtp": True, "num-mtp": 5} diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py b/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py index 4c5fa24ae4..ea700316b6 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py @@ -70,6 +70,11 @@ def golden_dir(tmp_path: Path) -> Path: "muse-spark-open", 3.4, ), + ( + "deepseek-v4.1-flash__dspark__native__probabilistic-block.yaml", + "deepseek-v4.1-flash", + 3.3, + ), ]: (directory / filename).write_text( yaml.safe_dump( @@ -573,6 +578,16 @@ def test_tilert_plan_uses_caller_decode_depth_and_keeps_prefill_real( } +def test_tilert_dsv41_profile_selects_the_dspark_curve(tmp_path: Path, golden_dir: Path) -> None: + recipe = tilert_recipe() + recipe["roles"]["decode"]["args"]["model"] = "dsv41" + path = tmp_path / "tilert.yaml" + path.write_text(yaml.safe_dump(recipe)) + env = {**ENV, "MODEL_PREFIX": "dsv41flash"} + commands = plan_commands(str(path), "tilert", [], env, golden_dir=golden_dir) + assert [command.acceptance_length for command in commands] == [3.3] + + @pytest.mark.parametrize( "environment", [{"EVAL_ONLY": "true"}, {"IS_AGENTIC": "0"}, {"SPEC_DECODING": "none"}, {}], diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 3473c978b2..3b27ea0c9c 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12407,6 +12407,17 @@ - "Remove TP1/EP1 at concurrency 1 and 4 and replace the search space with TP1/EP1 at concurrency 8, TP2/EP1 at concurrency 8, and TP1/EP1 with HiCache at concurrency 16, 24 and 32; the HiCache points use fp8_e4m3 kv-cache-dtype, and concurrency 16 and 24 use 2 speculative steps instead of 3." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4850 +- config-keys: + - dsv41flash-fp4-mi355x-tilert-agentic-dspark + scenario-type: + - agentic-coding + description: + - "Add DeepSeek-V4.1-Flash FP4 AgentX on MI355X with a disaggregated TileRT setup at concurrency 1: one vLLM ROCm prefill (ghcr.io/tile-ai/tilert-rocm-prefill:0.1.7: the validated MI355X DeepSeek-V4.1-Flash vLLM nightly e68efc68 plus the multi-protocol Mooncake build and ionic verbs provider of the TileRT ROCm decode image; TP4, vLLM's Rust front end, CUDA graphs up to 8192-token steps) sends the prompt KV over Mooncake to one TileRT decode (ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 with tilert 0.1.7.post2; TP8, five drafts from the checkpoint's own DSpark head) behind the TileRT PD router, with the full 1M context on both ranks and no context cap on the replay." + - "Throughput runs simulate the DeepSeek-V4.1-Flash DSpark golden AL at five drafts (3.51): a TileRT decode now takes its draft method from its model profile, and the dsv41 profile maps to dspark." + - "The TileRT router renders the chat template and tokenizes each request itself, so the prefill receives token ids: the model's chat template, adapted to render exactly what the prefill's vLLM deepseek_v41 tokenizer mode renders, ships as configs/dsv41flash-tilert-chat-template.jinja. tilert 0.1.7.post1 lets the router do so on the prefill's node; the prefill's flags match the other DeepSeek-V4.1-Flash configs." + - "tilert 0.1.7.post2 cuts the time to first token on long agentic contexts: the router asks the prefill for token ids per request (so the prefill runs vLLM's Rust front end) and reuses the token ids of the conversation prefix it has already rendered; the decode imports only the KV past the prefix it already holds from the previous turn, checked byte for byte against what it received then; the prefill stages the KV for the decode in a few vectorised copies." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4756 + - config-keys: - kimik3.1-fp4-mi355x-sglang-agentic-dflash scenario-type: From 1be9383a1c6b66032760ec9d16fae26fbc2a8690 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 15:26:44 -0500 Subject: [PATCH 033/111] chore(ubenchx): add trainium directory (#4905) --- ubenchX/trainium/.gitkeep | 1 + 1 file changed, 1 insertion(+) create mode 100644 ubenchX/trainium/.gitkeep diff --git a/ubenchX/trainium/.gitkeep b/ubenchX/trainium/.gitkeep new file mode 100644 index 0000000000..8b13789179 --- /dev/null +++ b/ubenchX/trainium/.gitkeep @@ -0,0 +1 @@ + From 518430e743a27831ff39b6bf3cb595136929482c Mon Sep 17 00:00:00 2001 From: Wenyao Gao <105094497+edwingao28@users.noreply.github.com> Date: Sat, 7 Nov 2026 12:38:21 -0800 Subject: [PATCH 034/111] =?UTF-8?q?[VideoGenX]=20include=20video=20configs?= =?UTF-8?q?=20in=20local=20change=20detection=20and=20rendering=20/=20[Vid?= =?UTF-8?q?eoGenX]=20=E6=9C=AC=E5=9C=B0=E5=8F=98=E6=9B=B4=E6=A3=80?= =?UTF-8?q?=E6=B5=8B=E4=B8=8E=E6=B8=B2=E6=9F=93=E7=BA=B3=E5=85=A5=E8=A7=86?= =?UTF-8?q?=E9=A2=91=E9=85=8D=E7=BD=AE=20(#4886)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test: cover video keys in change detection and recipe rendering * fix: include video scenarios in change detection and recipe rendering * test: write a fixture video spec instead of copying h3-v2.yaml * fix: keep sign-off matrix generation LLM-only * fix: record a base video bind failure per key in changed --- inferencex-e2e/infx/matrix/changed.py | 45 +++++++--- inferencex-e2e/infx/matrix/generate.py | 5 +- inferencex-e2e/infx/matrix/plan.py | 2 +- inferencex-e2e/infx/srt_slurm/generate.py | 6 +- .../infx/tests/matrix/test_changed.py | 90 ++++++++++++------- inferencex-e2e/infx/tests/srt_recipes.py | 77 +++++++++++++++- .../infx/tests/srt_slurm/test_generate.py | 46 ++++++++-- .../tests/workflows/test_signoff_facts.py | 49 ++++++++++ 8 files changed, 258 insertions(+), 62 deletions(-) diff --git a/inferencex-e2e/infx/matrix/changed.py b/inferencex-e2e/infx/matrix/changed.py index 329b9ccb4e..bcb971b536 100644 --- a/inferencex-e2e/infx/matrix/changed.py +++ b/inferencex-e2e/infx/matrix/changed.py @@ -14,8 +14,8 @@ from infx.config import MASTER_CONFIGS, RUNNER_CONFIG from .fingerprint import recipe_fingerprint, row_fingerprint -from .generate import generate_config_matrix -from .revision import snapshot +from .generate import SCENARIO_TYPES, generate_config_matrix +from .revision import Revision, snapshot from .validation import config_root, load_config_files, load_runner_file Matrix = dict[str, tuple[dict[tuple[str, int], dict[str, Any]], list[str]]] @@ -43,17 +43,33 @@ def _points(rows: list[dict], root: Path) -> tuple[dict, list[str]]: return points, errors +def config_points(key: str, master: dict, runners: dict, root: Path) -> tuple[dict, list[str]]: + """Every point of config ``key``, across the scenario types a changelog entry can select.""" + rows = generate_config_matrix( + [key], master, runners, scenario_types=SCENARIO_TYPES, eval_mode="none", root=root + ) + return _points(rows, root) + + def head_matrix(config_files: list[str], runner_config: str) -> Matrix: """Generate the working tree's matrix for each config key, as points and fingerprint errors.""" master = load_config_files(config_files) runners = load_runner_file(runner_config) root = config_root(config_files) - return { - key: _points( - generate_config_matrix([key], master, runners, eval_mode="none", root=root), root - ) - for key in master - } + return {key: config_points(key, master, runners, root) for key in master} + + +def _base_video_points(revision: Revision, key: str, single: str) -> tuple[dict, list[str]]: + """The video points of config ``key`` at ``revision``, or its generator's failure as an error. + + That generator exits when a video recipe cannot be bound, where head records the error per key. + """ + flags = ["--no-evals", "--scenario-type", "video-generation"] + try: + rows = revision.generate([key], flags, [single]) + except subprocess.CalledProcessError as error: + return {}, [error.stderr.strip().rsplit("\n", 1)[-1]] + return _points(rows, revision.root) def base_matrix(ref: str, config_files: list[str]) -> Matrix: @@ -72,10 +88,15 @@ def base_matrix(ref: str, config_files: list[str]) -> Matrix: ([key], ["--no-evals"], [single]) for key, single in zip(master, singles, strict=True) ] rows = revision.generate_each(requests, os.cpu_count() or 1) - return { - key: _points(generated, revision.root) - for key, generated in zip(master, rows, strict=True) - } + matrix: Matrix = {} + for (key, entry), single, generated in zip(master.items(), singles, rows, strict=True): + points, errors = _points(generated, revision.root) + if "video-generation" in entry.get("scenarios", {}): + video_points, video_errors = _base_video_points(revision, key, single) + points.update(video_points) + errors.extend(video_errors) + matrix[key] = (points, errors) + return matrix def _sorted(points: Iterable[dict]) -> list[dict]: diff --git a/inferencex-e2e/infx/matrix/generate.py b/inferencex-e2e/infx/matrix/generate.py index 52f4aaad53..8883945c37 100644 --- a/inferencex-e2e/infx/matrix/generate.py +++ b/inferencex-e2e/infx/matrix/generate.py @@ -30,6 +30,9 @@ seq_len_stoi = {"1k1k": (1024, 1024), "8k1k": (8192, 1024)} +# Every scenario type a changelog entry can select; unscoped generation leaves video out. +SCENARIO_TYPES = ("fixed-seq-len", "agentic-coding", "video-generation") + MIN_EVAL_CONC = 16 DEFAULT_EVAL_FRAMEWORK = "lm-eval" AUTOMATIC_AGENTIC_VENDOR_EVALS = { @@ -1797,7 +1800,7 @@ def main() -> list[dict]: parent_parser.add_argument( "--scenario-type", nargs="+", - choices=["fixed-seq-len", "agentic-coding", "video-generation"], + choices=list(SCENARIO_TYPES), required=False, help="Scenario type(s) to include. If not specified, fixed-seq-len and agentic-coding are generated; video-generation runs only when named.", ) diff --git a/inferencex-e2e/infx/matrix/plan.py b/inferencex-e2e/infx/matrix/plan.py index f03b731a45..23ace5ed8a 100644 --- a/inferencex-e2e/infx/matrix/plan.py +++ b/inferencex-e2e/infx/matrix/plan.py @@ -11,6 +11,7 @@ from .fingerprint import recipe_fingerprint from .generate import ( + SCENARIO_TYPES, EvalMode, drop_app_colliding_evals, freeze_config_value, @@ -27,7 +28,6 @@ load_runner_file, ) -SCENARIO_TYPES = ("fixed-seq-len", "agentic-coding", "video-generation") # An entry without scenario-type reruns these; video runs only for entries that name it. DEFAULT_SCENARIO_TYPES = ("fixed-seq-len", "agentic-coding") diff --git a/inferencex-e2e/infx/srt_slurm/generate.py b/inferencex-e2e/infx/srt_slurm/generate.py index b82de73a20..05d9c61df8 100644 --- a/inferencex-e2e/infx/srt_slurm/generate.py +++ b/inferencex-e2e/infx/srt_slurm/generate.py @@ -19,7 +19,7 @@ from infx.launch.drivers.srt import config, lanes, power from infx.launch.policy import launch_path from infx.launch.request import MultiNodeRequest -from infx.matrix.generate import expand_config_keys, generate_config_matrix +from infx.matrix.generate import SCENARIO_TYPES, expand_config_keys, generate_config_matrix from infx.matrix.validation import config_root, load_config_files, load_runner_file from infx.srt_slurm.single_node import select_recipe from infx.srt_slurm.synthetic_acceptance import build_overrides, selected_recipes @@ -292,7 +292,9 @@ def generate_recipes( recipes: dict[str, dict[str, Any]] = {} records = [] for key in expand_config_keys(config_keys, master): - points = generate_config_matrix([key], master, runners, eval_mode="none", root=root) + points = generate_config_matrix( + [key], master, runners, scenario_types=SCENARIO_TYPES, eval_mode="none", root=root + ) points = [point for point in points if point.get("srt-recipe")] if not points: raise ValueError(f"{key} has no srt-slurm points") diff --git a/inferencex-e2e/infx/tests/matrix/test_changed.py b/inferencex-e2e/infx/tests/matrix/test_changed.py index 14a09273ed..62501403da 100644 --- a/inferencex-e2e/infx/tests/matrix/test_changed.py +++ b/inferencex-e2e/infx/tests/matrix/test_changed.py @@ -9,26 +9,23 @@ import yaml from infx.matrix import changed -from infx.tests.srt_recipes import single_node_fragment, write_shared_blocks +from infx.tests.srt_recipes import ( + fixed_seq_entry, + single_node_fragment, + video_entry, + video_fragment, + write_shared_blocks, + write_video_specs, +) ROOT = Path(__file__).resolve().parents[3] RECIPES = "benchmarks/single_node/srt-slurm-recipes/fixture" -def entry(recipe: str, concurrencies: list[int]) -> dict: - return { - "image": "example/image:1", "model": "org/model", "model-prefix": "dsr1", - "precision": "fp8", "framework": "sglang", "runner": "cluster:fixture", - "multinode": False, "srt-recipe-dir": "fixture", - "scenarios": {"fixed-seq-len": [{"isl": 1024, "osl": 128, "search-space": [ - {"tp": 4, "conc-list": concurrencies, "srt-recipe": recipe}, - ]}]}, - } # fmt: skip - - -def write(root: Path, master: dict, edited: dict) -> None: - (root / "configs/nvidia-master.yaml").write_text(yaml.safe_dump(master)) - (root / RECIPES / "edited.yaml").write_text(yaml.safe_dump(edited)) +def write(project: Path, master: dict, edited: dict, video: dict) -> None: + (project / "configs/nvidia-master.yaml").write_text(yaml.safe_dump(master)) + (project / RECIPES / "edited.yaml").write_text(yaml.safe_dump(edited)) + (project / RECIPES / "video.yaml").write_text(yaml.safe_dump(video)) def git(root: Path, *args: str) -> str: @@ -38,47 +35,72 @@ def git(root: Path, *args: str) -> str: def test_keys_with_changed_recipes_or_points_are_listed(tmp_path, monkeypatch, capsys): - shutil.copytree(ROOT / "infx", tmp_path / "infx", ignore=shutil.ignore_patterns("__pycache__")) - (tmp_path / RECIPES).mkdir(parents=True) - write_shared_blocks(tmp_path) - (tmp_path / "configs/amd-master.yaml").write_text("{}\n") - (tmp_path / "configs/runners.yaml").write_text(yaml.safe_dump({ + project = tmp_path / "inferencex-e2e" + shutil.copytree(ROOT / "infx", project / "infx", ignore=shutil.ignore_patterns("__pycache__")) + (project / RECIPES).mkdir(parents=True) + write_shared_blocks(project) + write_video_specs(tmp_path) + (project / "configs/amd-master.yaml").write_text("{}\n") + (project / "configs/runners.yaml").write_text(yaml.safe_dump({ "labels": {"cluster:fixture": ["node-a"]}, "clusters": {"fixture": {"gpus-per-node": 8, "arch": "x86_64", "scheduler": "slurm", "slurm": {"partition": "batch", "exclusive": True}}}, })) # fmt: skip - (tmp_path / RECIPES / "stable.yaml").write_text(yaml.safe_dump(single_node_fragment(4))) - write(tmp_path, { - "edited": entry("edited.yaml", [2, 4]), "grown": entry("stable.yaml", [2]), - "same": entry("stable.yaml", [8]), "gone": entry("stable.yaml", [2]), - }, single_node_fragment(4)) # fmt: skip + (project / RECIPES / "stable.yaml").write_text(yaml.safe_dump(single_node_fragment(4))) + unbound = video_fragment() + del unbound["services"][0]["env"]["VIDEO_RUNTIME_REVISION"] + (project / RECIPES / "unbound.yaml").write_text(yaml.safe_dump(unbound)) + write(project, { + "edited": fixed_seq_entry("edited.yaml", [2, 4]), + "grown": fixed_seq_entry("stable.yaml", [2]), + "same": fixed_seq_entry("stable.yaml", [8]), + "video": video_entry(), + "unbound": video_entry("unbound.yaml"), + "gone": fixed_seq_entry("stable.yaml", [2]), + }, single_node_fragment(4), video_fragment()) # fmt: skip + (tmp_path / ".gitignore").write_text("__pycache__/\n") git(tmp_path, "init", "-q") git(tmp_path, "config", "user.name", "Test") git(tmp_path, "config", "user.email", "test@example.com") git(tmp_path, "add", ".") git(tmp_path, "commit", "-qm", "base") git(tmp_path, "tag", "base") - write(tmp_path, { - "edited": entry("edited.yaml", [2, 4]), "grown": entry("stable.yaml", [2, 4]), - "same": entry("stable.yaml", [8]), - }, single_node_fragment(4, **{"max-running-requests": 16})) # fmt: skip - monkeypatch.chdir(tmp_path) + write(project, { + "edited": fixed_seq_entry("edited.yaml", [2, 4]), + "grown": fixed_seq_entry("stable.yaml", [2, 4]), + "same": fixed_seq_entry("stable.yaml", [8]), + "video": video_entry(), + "unbound": video_entry("unbound.yaml"), + }, single_node_fragment(4, **{"max-running-requests": 16}), video_fragment(**{"max-running-requests": "7"})) # fmt: skip + (project / RECIPES / "unbound.yaml").write_text(yaml.safe_dump(video_fragment())) + monkeypatch.chdir(project) changed.main(["--base", "base", "--json"]) report = json.loads(capsys.readouterr().out) - assert (report["config-keys"], report["errors"]) == (4, []) + assert report["config-keys"] == 6 + [error] = report["errors"] + assert (error["config-key"], error["revision"]) == ("unbound", "base") + assert error["error"].endswith("video service requires a pinned VIDEO_RUNTIME_REVISION") assert [ (change["config-key"], change["base-points"], change["head-points"], [point["conc"] for point in change["added"]], [point["conc"] for point in change["removed"]]) for change in report["changed"] - ] == [("edited", 2, 2, [2, 4], [2, 4]), ("grown", 1, 2, [4], []), ("gone", 1, 0, [], [2])] # fmt: skip + ] == [ + ("edited", 2, 2, [2, 4], [2, 4]), ("grown", 1, 2, [4], []), ("unbound", 0, 1, [1], []), + ("video", 1, 1, [1], [1]), ("gone", 1, 0, [], [2]), + ] # fmt: skip assert git(tmp_path, "status", "--porcelain").splitlines() == [ - f" M {RECIPES}/edited.yaml", " M configs/nvidia-master.yaml", + f" M inferencex-e2e/{RECIPES}/edited.yaml", + f" M inferencex-e2e/{RECIPES}/unbound.yaml", + f" M inferencex-e2e/{RECIPES}/video.yaml", + " M inferencex-e2e/configs/nvidia-master.yaml", ] changed.main(["--base", "base"]) - assert capsys.readouterr().out.splitlines()[-1] == "3 of 4 config keys changed since base." + lines = capsys.readouterr().out.splitlines() + assert lines[-2] == "5 of 6 config keys changed since base." + assert lines[-1].startswith("warning: 1 rows of unbound have no fingerprint at base: ") def test_rows_without_fingerprints_are_compared_individually(tmp_path): diff --git a/inferencex-e2e/infx/tests/srt_recipes.py b/inferencex-e2e/infx/tests/srt_recipes.py index c01f9cc60c..fa954a8e6d 100644 --- a/inferencex-e2e/infx/tests/srt_recipes.py +++ b/inferencex-e2e/infx/tests/srt_recipes.py @@ -1,11 +1,19 @@ """Launchable srt-slurm recipes for tests that plan or fingerprint matrix rows.""" +import shutil from pathlib import Path import yaml +from infx.matrix.validation import VIDEO_SPEC_DIR from infx.srt_slurm.workload import NATIVE_TELEMETRY_BLOCK, SHARED_BLOCKS, TELEMETRY_BLOCK, VIDEO_BLOCK +ROOT = Path(__file__).resolve().parents[2] +VIDEO_MODEL = "example/video-model" +VIDEO_SPEC = "fixture.yaml" +VIDEO_SET = "fixture-set" +VIDEO_PORT = "30283" + def shared_blocks() -> dict[str, str]: """Each lane's shared block (its benchmark client) and the DCGM telemetry block, by path.""" @@ -21,9 +29,13 @@ def shared_blocks() -> dict[str, str]: blocks[NATIVE_TELEMETRY_BLOCK.as_posix()] = yaml.safe_dump( {"telemetry": {"enabled": True, "required": False}} ) - blocks[VIDEO_BLOCK.as_posix()] = yaml.safe_dump( - {"frontend": {"type": "none"}, "benchmark": {"type": "custom", "command": "python3 -m infx.bench video client"}} - ) + blocks[VIDEO_BLOCK.as_posix()] = yaml.safe_dump({ + "frontend": {"type": "none"}, + "benchmark": { + "type": "custom", "command": "python3 -m infx.bench video client --request-seconds 60", + "env": {"VIDEO_SERVER_SERVICE": "video-server", "VIDEO_SERVER_PORT": VIDEO_PORT}, + }, + }) # fmt: skip return blocks @@ -33,8 +45,67 @@ def write_shared_blocks(project: Path) -> None: (project / path).write_text(text) +def write_video_specs(repository: Path) -> None: + """Put the workload helper and a one-case spec serving ``VIDEO_MODEL`` beside the project.""" + specs = repository / VIDEO_SPEC_DIR + specs.mkdir(parents=True, exist_ok=True) + shutil.copy(ROOT.parent / VIDEO_SPEC_DIR / "video_spec.py", specs / "video_spec.py") + (specs / VIDEO_SPEC).write_text(yaml.safe_dump({ + "schema_version": 1, + "model": {"id": VIDEO_MODEL, "revision": "a" * 40}, + "generation": {"num_inference_steps": 1}, + "prompts": {"fixture": {"text": "a fixture prompt", "requires_motion": False, "requires_sound": False}}, + "sets": {VIDEO_SET: { + "plan_id": "fixture-plan", "repetitions": 1, "warmup_runs": 0, + "cases": [{"case_id": "fixture", "prompt": "fixture", "seed": 1}], + }}, + "reference": {}, + })) # fmt: skip + + +def fixed_seq_entry(recipe: str, concurrencies: list[int]) -> dict: + """A master entry with one TP4 fixed-sequence point per concurrency, served by ``recipe``.""" + return { + "image": "example/image:1", "model": "org/model", "model-prefix": "dsr1", + "precision": "fp8", "framework": "sglang", "runner": "cluster:fixture", + "multinode": False, "srt-recipe-dir": "fixture", + "scenarios": {"fixed-seq-len": [{"isl": 1024, "osl": 128, "search-space": [ + {"tp": 4, "conc-list": concurrencies, "srt-recipe": recipe}, + ]}]}, + } # fmt: skip + + +def video_entry(recipe: str = "video.yaml") -> dict: + """A master entry with one 2-GPU video point of ``VIDEO_SPEC``'s set served by ``recipe``.""" + return { + "image": "example/video:1", "model": VIDEO_MODEL, "model-prefix": "h3", + "precision": "bf16", "framework": "sglang", "runner": "cluster:fixture", + "multinode": False, "srt-recipe-dir": "fixture", + "scenarios": {"video-generation": [{"spec": VIDEO_SPEC, "set": VIDEO_SET, "search-space": [ + {"gpus": 2, "conc-list": [1], "srt-recipe": recipe}, + ]}]}, + } # fmt: skip + + def single_node_fragment(tp: int, **args) -> dict: """A single-node sglang fragment serving TP ``tp`` points without speculation.""" return {"schema": 2, "engine": "sglang", "roles": {"agg": { "nodes": 1, "workers": 1, "gpus": tp, "args": {"tensor-parallel-size": tp, **args}, }}} # fmt: skip + + +def video_fragment(**args: str) -> dict: + """A native sglang video fragment the binder accepts, with extra engine ``args`` appended.""" + return {"schema": 2, "name": "video-fixture", "resources": {"gpu_type": "fixture"}, "services": [{ + "name": "video-server", "type": "generic", "nodes": 1, "critical": True, + "command": ["python3", "-m", "infx.bench", "video", "serve"], + "args": [ + "--model-view", "snapshot", "--", "python3", "-m", "evaluator.srt_selected", "serve", + "--model-path", "{model_path}", "--model-id", "{model_id}", + "--revision", "{model_revision}", "--num-gpus", "{gpus_per_node}", + "--disable-conditioning-cache", "--host", "0.0.0.0", "--port", VIDEO_PORT, + *(f"--{flag}={value}" for flag, value in args.items()), + ], + "env": {"VIDEO_SERVER_PORT": VIDEO_PORT, "VIDEO_RUNTIME_REVISION": "b" * 40}, + "readiness": {"http": {"port": int(VIDEO_PORT), "path": "/health"}}, + }]} # fmt: skip diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_generate.py b/inferencex-e2e/infx/tests/srt_slurm/test_generate.py index d5751eecba..d301b08b40 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_generate.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_generate.py @@ -9,6 +9,14 @@ from infx.cli import main from infx.launch.drivers.srt import lanes from infx.launch.policy import LaunchPath +from infx.tests.srt_recipes import ( + VIDEO_MODEL, + VIDEO_SPEC, + video_entry, + video_fragment, + write_shared_blocks, + write_video_specs, +) ROOT = Path(__file__).resolve().parents[3] FRAGMENT = { @@ -51,6 +59,7 @@ "power": True, }]}]}, }, + "video": video_entry(), } # fmt: skip SHARED = { "telemetry-native.yaml": {"telemetry": {"enabled": True, "required": False}}, @@ -64,11 +73,14 @@ @pytest.fixture def project(tmp_path): + """The end-to-end project beside the video harness, as the repository lays them out.""" + project = tmp_path / "inferencex-e2e" + write_shared_blocks(project) + write_video_specs(tmp_path) for name, block in SHARED.items(): - (tmp_path / "configs/srt-recipes" / name).parent.mkdir(parents=True, exist_ok=True) - (tmp_path / "configs/srt-recipes" / name).write_text(yaml.safe_dump(block)) - (tmp_path / "configs/master.yaml").write_text(yaml.safe_dump(MASTER)) - (tmp_path / "configs/runners.yaml").write_text(yaml.safe_dump({ + (project / "configs/srt-recipes" / name).write_text(yaml.safe_dump(block)) + (project / "configs/master.yaml").write_text(yaml.safe_dump(MASTER)) + (project / "configs/runners.yaml").write_text(yaml.safe_dump({ "labels": {"fixture": ["fixture_0"], "cluster:fixture": ["fixture_0"]}, "clusters": {"fixture": {"gpus-per-node": 8, "arch": "x86_64", "scheduler": "slurm", "slurm": { "partition": "p", "exclusive": True, @@ -88,6 +100,7 @@ def project(tmp_path): **RECIPE, "frontend": {"type": "sglang", "enable_multiple_frontends": False}, "roles": {"agg": {"nodes": 1, "workers": 1, "gpus": 4, "args": {"tensor-parallel-size": 4}}}, }, + "single_node/srt-slurm-recipes/fixture/video.yaml": video_fragment(), "multi_node/srt-slurm-recipes/fixture/agentic.yaml": { **RECIPE, "frontend": {"type": "sglang-router"}, "roles": { role: {"nodes": 1, "workers": 1, "gpus": 8, "args": {"tensor-parallel-size": 8}} @@ -96,11 +109,11 @@ def project(tmp_path): }, } # fmt: skip for path, data in recipes.items(): - (tmp_path / "benchmarks" / path).parent.mkdir(parents=True, exist_ok=True) - (tmp_path / "benchmarks" / path).write_text(yaml.safe_dump(data)) - (tmp_path / "utils/srt-slurm").mkdir(parents=True) - (tmp_path / "utils/srt-slurm/src").symlink_to(ROOT / "utils/srt-slurm/src") - return tmp_path + (project / "benchmarks" / path).parent.mkdir(parents=True, exist_ok=True) + (project / "benchmarks" / path).write_text(yaml.safe_dump(data)) + (project / "utils/srt-slurm").mkdir(parents=True) + (project / "utils/srt-slurm/src").symlink_to(ROOT / "utils/srt-slurm/src") + return project @pytest.fixture @@ -314,6 +327,21 @@ def test_output_must_be_new_or_empty(project, tmp_path, capsys): assert "Output directory must be empty" in capsys.readouterr().err +def test_a_video_point_renders_its_bound_server_recipe(project, tmp_path): + output = tmp_path / "out" + assert generate(project, output, "video") == 0 + + [(record, recipe)] = written(output) + assert (record["variant"], record["cluster"]) == (None, "fixture") + assert (record["matrix"]["scenario-type"], record["matrix"]["conc-list"]) == ("video-generation", [1]) + assert recipe["model"] == {"path": f"hf:{VIDEO_MODEL}", "container": "example/video:1", "precision": "bf16"} + assert recipe["resources"]["gpus_per_node"] == 2 + [server] = recipe["services"] + assert server["args"][server["args"].index("--num-gpus") + 1] == "2" + assert server["env"]["VIDEO_SPEC_PATH"] == f"/video-package/specs/{VIDEO_SPEC}" + assert recipe["benchmark"]["env"]["CONC_LIST"] == "1" + + def test_a_recipe_srtctl_rejects_fails_without_writing(project, tmp_path, capsys): recipe = project / "benchmarks/single_node/srt-slurm-recipes/fixture/recipe.yaml" recipe.write_text(yaml.safe_dump({**FRAGMENT, "base": {**FRAGMENT["base"], "no_such_field": 1}})) diff --git a/inferencex-e2e/infx/tests/workflows/test_signoff_facts.py b/inferencex-e2e/infx/tests/workflows/test_signoff_facts.py index 21c154d75b..33826373c3 100644 --- a/inferencex-e2e/infx/tests/workflows/test_signoff_facts.py +++ b/inferencex-e2e/infx/tests/workflows/test_signoff_facts.py @@ -1,10 +1,21 @@ """Exercise the verifier's deterministic facts against controlled inputs.""" import json +import subprocess import pytest +import yaml from infx.klaud.models import Feed +from infx.matrix.validation import VIDEO_SPEC_DIR +from infx.tests.srt_recipes import ( + VIDEO_SPEC, + fixed_seq_entry, + single_node_fragment, + video_entry, + video_fragment, + write_shared_blocks, +) from infx.workflows import signoff_facts SERIES = ["--model", "m", "--precision", "fp4", "--framework", "sglang", "--hardware", "b200"] @@ -268,3 +279,41 @@ def test_eval_gap_estimates_zero_standard_errors_from_sample_size(): def test_digest_pins_are_inspected_by_digest(monkeypatch, image, inspected): monkeypatch.setattr(signoff_facts, "_inspect", lambda reference: {"inspected": reference}) assert signoff_facts.inspect_image(image)["inspected"] == inspected + + +def test_matrix_at_generates_llm_points_only_and_runs_no_revision_code(tmp_path, monkeypatch): + project = tmp_path / "inferencex-e2e" + recipes = project / "benchmarks/single_node/srt-slurm-recipes/fixture" + write_shared_blocks(project) + (project / "configs/runners.yaml").write_text(yaml.safe_dump({ + "labels": {"cluster:fixture": ["node-a"]}, + "clusters": {"fixture": {"gpus-per-node": 8, "arch": "x86_64", "scheduler": "slurm", + "slurm": {"partition": "batch", "exclusive": True}}}, + })) # fmt: skip + (project / "configs/nvidia-master.yaml").write_text( + yaml.safe_dump({"llm": fixed_seq_entry("stable.yaml", [2]), "video": video_entry()}) + ) + (project / "configs/amd-master.yaml").write_text("{}\n") + recipes.mkdir(parents=True) + (recipes / "stable.yaml").write_text(yaml.safe_dump(single_node_fragment(4))) + (recipes / "video.yaml").write_text(yaml.safe_dump(video_fragment())) + # git archive rejects a DATA_PATHS entry that the tree lacks. + (project / "benchmarks/multi_node/srt-slurm-recipes").mkdir(parents=True) + (project / "benchmarks/multi_node/srt-slurm-recipes/.keep").touch() + specs = tmp_path / VIDEO_SPEC_DIR + specs.mkdir(parents=True) + (specs / "video_spec.py").write_text('raise RuntimeError("video_spec.py from the revision tree ran")\n') + (specs / VIDEO_SPEC).write_text("{}\n") + for args in (["init", "-q"], ["add", "."], ["commit", "-qm", "base"]): + subprocess.run( + ["git", "-c", "user.name=t", "-c", "user.email=t@example.com", *args], + cwd=tmp_path, check=True, capture_output=True, + ) # fmt: skip + monkeypatch.chdir(tmp_path) + + matrix, errors = signoff_facts.matrix_at("HEAD", tmp_path / "work") + + assert errors == {} + assert { + key: ([conc for _, conc in points], messages) for key, (points, messages) in matrix.items() + } == {"llm": ([2], []), "video": ([], [])} From ac95198dece056f6675ff6f60b6b6fc038aa6956 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:02:22 -0500 Subject: [PATCH 035/111] refactor(ubenchX): move sm_l2_distance to l2_distance/nvidia (#4910) Make room for per-vendor L2 distance benchmarks under ubenchX/l2_distance/. The CUDA benchmark and its B200 results move unchanged; only the README table rows and the docstring usage paths change. Co-authored-by: Claude Fable 5.1 --- ubenchX/README.md | 2 +- ubenchX/README_zh.md | 2 +- ubenchX/{sm_l2_distance => l2_distance/nvidia}/Makefile | 0 ubenchX/{sm_l2_distance => l2_distance/nvidia}/bench.py | 4 ++-- .../nvidia}/l2_pointer_chase.cu | 0 .../{sm_l2_distance => l2_distance/nvidia}/results/.gitkeep | 0 .../nvidia}/results/b200_2026-10-09.csv | 0 .../nvidia}/results/b200_sm_info_2026-10-09.csv | 0 8 files changed, 4 insertions(+), 4 deletions(-) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/Makefile (100%) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/bench.py (96%) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/l2_pointer_chase.cu (100%) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/results/.gitkeep (100%) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/results/b200_2026-10-09.csv (100%) rename ubenchX/{sm_l2_distance => l2_distance/nvidia}/results/b200_sm_info_2026-10-09.csv (100%) diff --git a/ubenchX/README.md b/ubenchX/README.md index 1fb1ec0088..0f7c485c12 100644 --- a/ubenchX/README.md +++ b/ubenchX/README.md @@ -9,7 +9,7 @@ repository root in an environment with the required device libraries. |------|-------------|--------------| | [gemm](gemm/bench.py) | Dense matrix multiplication latency and TFLOP/s | PyTorch with CUDA/ROCm, or compatible `torch_npu` and CANN for Ascend | | [mem_bw](mem_bw/bench.py) | Device copy bandwidth, counting reads and writes | PyTorch with CUDA/ROCm and Triton | -| [sm_l2_distance](sm_l2_distance/bench.py) | Per-SM L2 pointer-chase latency differences | NVIDIA GPU and CUDA toolkit (`nvcc`) | +| [l2_distance/nvidia](l2_distance/nvidia/bench.py) | Per-SM L2 pointer-chase latency differences | NVIDIA GPU and CUDA toolkit (`nvcc`) | | [gpc_query](gpc_query/) | GPC/TPC configuration from cluster scheduling | NVIDIA GPU with `sm_90+` and CUDA toolkit | ## GEMM diff --git a/ubenchX/README_zh.md b/ubenchX/README_zh.md index f05152f85b..a01eff8f04 100644 --- a/ubenchX/README_zh.md +++ b/ubenchX/README_zh.md @@ -9,7 +9,7 @@ |------|----------|------| | [gemm](gemm/bench.py) | 稠密矩阵乘法的延迟和 TFLOP/s | 支持 CUDA/ROCm 的 PyTorch，或适配昇腾的兼容 `torch_npu` 与 CANN | | [mem_bw](mem_bw/bench.py) | 设备内存拷贝带宽，计入读取和写入 | 支持 CUDA/ROCm 的 PyTorch 与 Triton | -| [sm_l2_distance](sm_l2_distance/bench.py) | 各 SM 的 L2 指针追踪延迟差异 | NVIDIA GPU 与 CUDA 工具包（`nvcc`） | +| [l2_distance/nvidia](l2_distance/nvidia/bench.py) | 各 SM 的 L2 指针追踪延迟差异 | NVIDIA GPU 与 CUDA 工具包（`nvcc`） | | [gpc_query](gpc_query/) | 通过线程块集群调度查询 GPC/TPC 配置 | 支持 `sm_90+` 的 NVIDIA GPU 与 CUDA 工具包 | ## GEMM diff --git a/ubenchX/sm_l2_distance/Makefile b/ubenchX/l2_distance/nvidia/Makefile similarity index 100% rename from ubenchX/sm_l2_distance/Makefile rename to ubenchX/l2_distance/nvidia/Makefile diff --git a/ubenchX/sm_l2_distance/bench.py b/ubenchX/l2_distance/nvidia/bench.py similarity index 96% rename from ubenchX/sm_l2_distance/bench.py rename to ubenchX/l2_distance/nvidia/bench.py index 3cb0d0b04b..62b9e2cdd7 100644 --- a/ubenchX/sm_l2_distance/bench.py +++ b/ubenchX/l2_distance/nvidia/bench.py @@ -12,8 +12,8 @@ Requirements: CUDA toolkit (nvcc) with cooperative-groups support. Usage: - python ubenchX/sm_l2_distance/bench.py # auto-detect arch - python ubenchX/sm_l2_distance/bench.py --arch sm_100a --num-sms 148 + python ubenchX/l2_distance/nvidia/bench.py # auto-detect arch + python ubenchX/l2_distance/nvidia/bench.py --arch sm_100a --num-sms 148 Ported from SemiAnalysisAI/microbench-blackwell (sm_l2_distance/). """ diff --git a/ubenchX/sm_l2_distance/l2_pointer_chase.cu b/ubenchX/l2_distance/nvidia/l2_pointer_chase.cu similarity index 100% rename from ubenchX/sm_l2_distance/l2_pointer_chase.cu rename to ubenchX/l2_distance/nvidia/l2_pointer_chase.cu diff --git a/ubenchX/sm_l2_distance/results/.gitkeep b/ubenchX/l2_distance/nvidia/results/.gitkeep similarity index 100% rename from ubenchX/sm_l2_distance/results/.gitkeep rename to ubenchX/l2_distance/nvidia/results/.gitkeep diff --git a/ubenchX/sm_l2_distance/results/b200_2026-10-09.csv b/ubenchX/l2_distance/nvidia/results/b200_2026-10-09.csv similarity index 100% rename from ubenchX/sm_l2_distance/results/b200_2026-10-09.csv rename to ubenchX/l2_distance/nvidia/results/b200_2026-10-09.csv diff --git a/ubenchX/sm_l2_distance/results/b200_sm_info_2026-10-09.csv b/ubenchX/l2_distance/nvidia/results/b200_sm_info_2026-10-09.csv similarity index 100% rename from ubenchX/sm_l2_distance/results/b200_sm_info_2026-10-09.csv rename to ubenchX/l2_distance/nvidia/results/b200_sm_info_2026-10-09.csv From f7dfdf871d406ac1d1d291c5e923038a5fef2bfc Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 19:38:40 -0500 Subject: [PATCH 036/111] =?UTF-8?q?Tune=20B300=20GLM-5.3-Flash=20FP8=20Age?= =?UTF-8?q?ntX=20/=20=E8=B0=83=E4=BC=98=20B300=20GLM-5.3-Flash=20FP8=20Age?= =?UTF-8?q?ntX=20(#4902)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf: tune B300 GLM-5.3-Flash FP8 AgentX serving * chore: link B300 FP8 tuning changelog * fix: retain throughput when expanding B300 accuracy coverage --- .../sglang/b300-fp8-mtp/agentic.yaml | 22 +++++++++++++++++-- inferencex-e2e/configs/nvidia-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 10 +++++++++ 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml index 19db169d69..3b51eadd76 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml @@ -17,8 +17,6 @@ base: nodes: 1 workers: 1 gpus: 8 - # Serve flags follow the cookbook's B300 Low Latency cell at - # https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash. args: served-model-name: zai-org/GLM-5.3-Flash tensor-parallel-size: 8 @@ -50,6 +48,26 @@ base: AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' override_none: + roles: + agg: + gpus: 4 + args: + tensor-parallel-size: 4 + enable-linear-replayssm-spec: true + max-running-requests: 32 + cuda-graph-max-bs-decode: 32 + cuda-graph-backend-prefill: breakable + cuda-graph-max-bs-prefill: 8192 + mem-fraction-static: 0.9 + prefill-decode-interval: 1 + chunked-prefill-size: 16384 + mamba-max-states-per-path: 2 + mamba-full-memory-ratio: 0.5 + random-seed: 316023864 + weight-loader-prefetch-checkpoints: true + weight-loader-prefetch-num-threads: 16 + watchdog-timeout: 3600 + enable-cache-report: true benchmark: env: KV_OFFLOADING: none diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 857008156d..6ce35bcdef 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -7567,7 +7567,7 @@ glm5.3flash-fp8-b300-sglang-agentic-mtp: agentic-coding: - dram-utilization: 0.935 search-space: - - { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32, 48], srt-recipe: "agentic.yaml:override_none" } + - { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32, 48, 64], srt-recipe: "agentic.yaml:override_none" } - { tp: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64, 96], srt-recipe: "agentic.yaml:override_dram" } glm5.3flash-fp4-b300-sglang-agentic-mtp: diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 3b27ea0c9c..75b0ab1383 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12430,3 +12430,13 @@ - "Add concurrency 56." - "Concurrency 8 to 56: SGLANG_HICACHE_MAMBA_HOST_RATIO 12 -> 10; the image's larger device pools grew the pinned host pools past the node's GTT (hipHostMalloc failed at concurrency 32)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4846 + +- config-keys: + - glm5.3flash-fp8-b300-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Tune B300 GLM-5.3-Flash FP8 GPU-resident AgentX with TP4, prefill-decode interval 1, 16K prefill chunks, a two-state Mamba path cap, and Mamba/full memory ratio 0.5." + - "Enable ReplaySSM speculative verification, cap server admission and decode graphs at 32, and pin breakable prefill graphs to 8192 tokens with static memory fraction 0.9." + - "Add GPU-resident client concurrency 64 while preserving the TP8 HiCache points, full AgentX workload, measured MTP acceptance, and simulation-off accuracy evaluation." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4902 From daa4ee0ff06f267342f6628ba2b67e6c2ecca8ce Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 7 Nov 2026 21:37:41 -0500 Subject: [PATCH 037/111] Update README.md --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index fdf177b2cc..b9a9908b6e 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,6 @@ Trusted by Operators of Trillion Dollar Token Factories such as OpenAI, Meta, Mi - **[2026/11]** 🔥 Kimi K3.1 Pro: continuous benchmarks live since Day 0 on B200, B300, GB200 and MI355X (SGLang + vLLM) - **[2026/11]** 🔥 PowerX: measured power is now collected by default on multi-node runs, including NVL72 Grace CPU, per-host GPU and AMD EPYC CPU package power -- **[2026/11]** GLM-5.3-Flash: added AgentX benchmarks on GB200 (FP8 + NVFP4), MI300X and MI325X with HiCache [dashboard](https://inferencex.semianalysis.com/agentx) - **[2026/10]** MiniMax M3.1 Flash: Day 0 AgentX benchmarks with DSpark speculative decoding on B200 and B300 [dashboard](https://inferencex.semianalysis.com/agentx) - **[2026/10]** Muse Spark Open: added AgentX benchmarks on B300 and GB300 (SGLang) [dashboard](https://inferencex.semianalysis.com/agentx) - **[2026/10]** Open-source power model (beta) for estimating system power from measured GPU, CPU and memory telemetry From dedc9f6ba408e899a6188e39f48d38954c5a3b74 Mon Sep 17 00:00:00 2001 From: zijie Date: Sat, 7 Nov 2026 20:56:48 -0700 Subject: [PATCH 038/111] =?UTF-8?q?[AMD][MI35X]=201106=20Muse=20Spark=20sg?= =?UTF-8?q?lang=20dflash=20agentic=20/=20[AMD][MI35X]=201106=20Muse=20Spar?= =?UTF-8?q?k=20sglang=20dflash=20agentic=20=E9=85=8D=E7=BD=AE=E6=9B=B4?= =?UTF-8?q?=E6=96=B0=20(#4881)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [AMD][MI35X] 1106 Muse Spark sglang dflash agentic / [AMD][MI35X] 1106 Muse Spark sglang dflash agentic 配置更新 Update Muse Spark MI355X SGLang AgentX config for the 1106 image * Update PR number * Remove Muse Spark MI355X override rows no search-space row selects * Keep TEP4 at 28, 36 and 40; C36 prefill every 48 decode steps * Give the Muse Spark MI355X server warm-up 1800 s on a cold JIT cache * Pin the Muse Spark MI355X checkpoint revision --------- Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com> Co-authored-by: billishyahao --- .../sglang/mi355x-fp4-mtp/agentic.yaml | 147 +++++------------- inferencex-e2e/configs/amd-master.yaml | 6 +- inferencex-e2e/perf-changelog.yaml | 9 ++ 3 files changed, 49 insertions(+), 113 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/musespark/sglang/mi355x-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/musespark/sglang/mi355x-fp4-mtp/agentic.yaml index 1ea4e7f22b..514589cf74 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/musespark/sglang/mi355x-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/musespark/sglang/mi355x-fp4-mtp/agentic.yaml @@ -21,6 +21,7 @@ base: tensor-parallel-size: 8 expert-parallel-size: 8 served-model-name: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 + revision: c7bc74fd64371315a091bc15f9723882eead0dc1 attention-backend: aiter page-size: 16 kv-cache-dtype: fp8_e4m3 @@ -46,6 +47,7 @@ base: PYTHONNOUSERSITE: '1' # Outlast AIPerf's pooled connections past Uvicorn's keep-alive. SGLANG_TIMEOUT_KEEP_ALIVE: '900' + SGLANG_WARMUP_TIMEOUT: '1800' SGLANG_USE_AITER: '1' SGLANG_USE_AITER_UNIFIED_ATTN: '1' MUSE_MOE_BACKEND: flydsl @@ -59,32 +61,20 @@ base: AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' -override_tp8_c1: +override_tp8_c6: roles: agg: args: monokernel-backend: amd max-running-requests: 1 cuda-graph-max-bs-decode: 1 + stream-interval: 128 benchmark: env: - CONC: '1' + CONC: '6' KV_OFFLOADING: none -override_tp8_c4: - roles: - agg: - args: - monokernel-backend: amd - max-running-requests: 2 - cuda-graph-max-bs-decode: 2 - prefill-decode-interval: 64 - benchmark: - env: - CONC: '4' - KV_OFFLOADING: none - -override_tp8_c8: +override_tp8_c10: roles: agg: args: @@ -92,128 +82,59 @@ override_tp8_c8: max-running-requests: 2 cuda-graph-max-bs-decode: 2 prefill-decode-interval: 64 + swa-full-tokens-ratio: 0.3 + schedule-policy: shortest-prefill-first + stream-interval: 40 benchmark: env: - CONC: '8' + CONC: '10' KV_OFFLOADING: none -override_tp8_c13: +override_tp8_c28: roles: agg: args: - monokernel-backend: amd - max-running-requests: 2 - cuda-graph-max-bs-decode: 2 + max-running-requests: 8 + cuda-graph-max-bs-decode: 8 + chunked-prefill-size: 32512 + max-prefill-tokens: 32512 + mem-fraction-static: 0.88 prefill-decode-interval: 64 + schedule-policy: fcfs + stream-interval: 20 + min-free-slots-delay: 4 benchmark: env: - CONC: '13' + CONC: '28' KV_OFFLOADING: none -override_tp4_c10: - roles: - agg: - args: - tensor-parallel-size: 4 - expert-parallel-size: 4 - enable-hierarchical-cache: true - hicache-write-policy: write_through - hicache-io-backend: kernel - hicache-mem-layout: page_first - hicache-size: 253 - swa-full-tokens-ratio: 0.1 - max-running-requests: 4 - cuda-graph-max-bs-decode: 4 - chunked-prefill-size: 8192 - max-prefill-tokens: 8192 - prefill-decode-interval: 64 - mem-fraction-static: 0.88 - env: - PYTORCH_HIP_ALLOC_CONF: expandable_segments:True - PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True - gpus: 4 - benchmark: - env: - CONC: '10' - KV_OFFLOADING: dram - -override_tp4_c16: - roles: - agg: - args: - tensor-parallel-size: 4 - expert-parallel-size: 4 - enable-hierarchical-cache: true - hicache-write-policy: write_through - hicache-io-backend: kernel - hicache-mem-layout: page_first - hicache-size: 253 - swa-full-tokens-ratio: 0.1 - schedule-policy: shortest-prefill-first - max-running-requests: 12 - cuda-graph-max-bs-decode: 12 - chunked-prefill-size: 8192 - max-prefill-tokens: 8192 - prefill-decode-interval: 96 - env: - PYTORCH_HIP_ALLOC_CONF: expandable_segments:True - PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True - gpus: 4 - benchmark: - env: - CONC: '16' - KV_OFFLOADING: dram - -override_tp4_c24: - roles: - agg: - args: - tensor-parallel-size: 4 - expert-parallel-size: 4 - enable-hierarchical-cache: true - hicache-write-policy: write_through - hicache-io-backend: kernel - hicache-mem-layout: page_first - hicache-size: 253 - swa-full-tokens-ratio: 0.1 - schedule-policy: shortest-prefill-first - max-running-requests: 48 - chunked-prefill-size: 8192 - max-prefill-tokens: 8192 - prefill-decode-interval: 48 - mem-fraction-static: 0.96 - env: - PYTORCH_HIP_ALLOC_CONF: expandable_segments:True - PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True - gpus: 4 - benchmark: - env: - CONC: '24' - KV_OFFLOADING: dram - override_tp4_c28: roles: agg: args: tensor-parallel-size: 4 + page-size: 128 expert-parallel-size: 4 enable-hierarchical-cache: true hicache-write-policy: write_through hicache-io-backend: kernel hicache-mem-layout: page_first hicache-size: 310 - swa-full-tokens-ratio: 0.1 + swa-full-tokens-ratio: 0.3 schedule-policy: shortest-prefill-first max-running-requests: 56 cuda-graph-max-bs-decode: 56 chunked-prefill-size: 16384 max-prefill-tokens: 16384 prefill-decode-interval: 80 + stream-interval: 20 + scheduler-recv-interval: 1 mem-fraction-static: 0.86 env: PYTORCH_HIP_ALLOC_CONF: expandable_segments:True PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1' + SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: '1' gpus: 4 benchmark: env: @@ -225,19 +146,21 @@ override_tp4_c36: agg: args: tensor-parallel-size: 4 + page-size: 128 expert-parallel-size: 4 enable-hierarchical-cache: true hicache-write-policy: write_through hicache-io-backend: kernel hicache-mem-layout: page_first hicache-size: 310 - swa-full-tokens-ratio: 0.1 + swa-full-tokens-ratio: 0.2 schedule-policy: shortest-prefill-first max-running-requests: 72 cuda-graph-max-bs-decode: 72 chunked-prefill-size: 16384 max-prefill-tokens: 16384 - prefill-decode-interval: 24 + prefill-decode-interval: 48 + stream-interval: 20 mem-fraction-static: 0.86 env: PYTORCH_HIP_ALLOC_CONF: expandable_segments:True @@ -254,25 +177,29 @@ override_tp4_c40: agg: args: tensor-parallel-size: 4 + page-size: 128 expert-parallel-size: 4 enable-hierarchical-cache: true hicache-write-policy: write_through hicache-io-backend: kernel hicache-mem-layout: page_first hicache-size: 310 - mem-fraction-static: 0.86 - swa-full-tokens-ratio: 0.1 + swa-full-tokens-ratio: 0.2 schedule-policy: shortest-prefill-first max-running-requests: 80 + cuda-graph-max-bs-decode: 80 chunked-prefill-size: 16384 max-prefill-tokens: 16384 - prefill-decode-interval: 80 + prefill-decode-interval: 24 + stream-interval: 20 + mem-fraction-static: 0.86 env: - SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1' PYTORCH_HIP_ALLOC_CONF: expandable_segments:True PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True + SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1' gpus: 4 benchmark: env: CONC: '40' KV_OFFLOADING: dram + diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 17ed3e0570..86c68fcafd 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -1092,7 +1092,7 @@ dsv41flash-fp4-mi355x-sglang-agentic-dspark: - { tp: 2, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [32, 64], srt-recipe: agentic.yaml } musespark-fp4-mi355x-sglang-agentic-dflash: - image: lmsysorg/sglang-rocm:muse-spark-open-1027@sha256:7184d3ae2bfdc75a5bb8eb497b768e370ef7b213879cb4d97fbaaa3e8e5f5040 + image: lmsysorg/sglang-rocm:muse-spark-open-mi355-1106@sha256:d231181df079a5b271a0459d024492c84a36caf8cb26ee019abad29e17eea4d2 model: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 model-prefix: musespark runner: cluster:mi355x-amds @@ -1104,8 +1104,8 @@ musespark-fp4-mi355x-sglang-agentic-dflash: agentic-coding: - dram-utilization: 0.85 search-space: - - { tp: 8, ep: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 4, 8, 13], srt-recipe: agentic.yaml } - - { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [10, 16, 24, 28, 36, 40], srt-recipe: agentic.yaml } + - { tp: 8, ep: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [6, 10, 28], srt-recipe: agentic.yaml } + - { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [28, 36, 40], srt-recipe: agentic.yaml } kimik3.1-fp4-mi355x-sglang-agentic-dflash: image: lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:346a6d832bbfcb5776d8d938154870682a891278a8adc264ed996ccf3d3b6762 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 75b0ab1383..4d7d1d548a 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12440,3 +12440,12 @@ - "Enable ReplaySSM speculative verification, cap server admission and decode graphs at 32, and pin breakable prefill graphs to 8192 tokens with static memory fraction 0.9." - "Add GPU-resident client concurrency 64 while preserving the TP8 HiCache points, full AgentX workload, measured MTP acceptance, and simulation-off accuracy evaluation." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4902 + +- config-keys: + - musespark-fp4-mi355x-sglang-agentic-dflash + scenario-type: + - agentic-coding + description: + - "Bump to lmsysorg/sglang-rocm:muse-spark-open-mi355-1106." + - "Concurrency 6, 10 and 28: TP8/EP8 with GPU-only KV; 28, 36 and 40: TP4/EP4 with a 310 GB-per-rank HiCache DRAM tier." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4881 From 6d26209ff0abda4023bfccdf6a024056ae29f8be Mon Sep 17 00:00:00 2001 From: Hyukjoon Lee Date: Sun, 8 Nov 2026 13:41:08 +0900 Subject: [PATCH 039/111] =?UTF-8?q?perf(amd):=20Kimi-K3.1=20MI355X=20vLLM?= =?UTF-8?q?=20agentic:=20per-conc=20max-num-seqs,=20conc=201=20DCP1,=20con?= =?UTF-8?q?c=204=20seven=20draft=20tokens=20/=20perf(amd):=20Kimi-K3.1=20M?= =?UTF-8?q?I355X=20vLLM=20agentic=EF=BC=9A=E6=8C=89=E5=B9=B6=E5=8F=91?= =?UTF-8?q?=E8=B0=83=E6=95=B4=20max-num-seqs=EF=BC=8C=E5=B9=B6=E5=8F=91=20?= =?UTF-8?q?1=20=E6=94=B9=E4=B8=BA=20DCP1=EF=BC=8C=E5=B9=B6=E5=8F=91=204=20?= =?UTF-8?q?=E4=BD=BF=E7=94=A8=207=20=E4=B8=AA=E8=8D=89=E7=A8=BF=20token=20?= =?UTF-8?q?(#4887)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(amd): Kimi-K3.1 MI355X vLLM agentic: per-conc max-num-seqs, conc 1 DCP1, conc 4 seven draft tokens Co-Authored-By: Claude Opus 5.5 * perf-changelog: link #4887 Co-Authored-By: Claude Opus 5.5 * Drop added comments Co-Authored-By: Claude Opus 5.5 * docs(changelog): sync Kimi-K3.1 MI355X vLLM AgentX block with the diff - The image stays vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336). - Concurrency 1 moves from TP8 x DCP8 to TP8 x DCP1 (master config dcp-size 8 -> 1; the recipe's c1 override drops decode-context-parallel-size, dcp-comm-backend and cp-kv-cache-interleave-size). It keeps GPU-resident KV, 7 draft tokens and the latency_specialized decode backend. Concurrency 4-48 stay TP8 x DCP8 with DRAM offload. - Concurrency 4: DSpark draft tokens 5 -> 7, so throughput runs use golden acceptance length 3.69 instead of 3.41. The decode capture sizes change from 6, 12 to 8, 16 to match max-num-seqs 2 x (1 + 7). - max-num-seqs at concurrency 10/16/24/32 changes from 6/10/16/24 to 7/8/13/20. Concurrency 10 adds capture size 42 (7 x (1 + 5)). Otherwise the draft lengths, capture lists, max-num-batched-tokens and gpu-memory-utilization at these points, and every setting at concurrency 48, are unchanged. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config names the draft and sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache stays FP8, the same as the target's. DCP1 at concurrency 1 and 7 draft tokens at concurrency 4 do not change these types. - 镜像保持为 vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1（vLLM 0.0.0+mum.14d1ec2336）。 - 并发 1 从 TP8 x DCP8 改为 TP8 x DCP1（master config 的 dcp-size 8 -> 1；recipe 的 c1 override 去掉 decode-context-parallel-size、dcp-comm-backend 和 cp-kv-cache-interleave-size）。它仍使用 GPU 常驻 KV、7 个草稿 token 和 latency_specialized decode 后端。并发 4-48 仍为 TP8 x DCP8 并使用 DRAM offload。 - 并发 4：DSpark 草稿 token 数 5 -> 7，因此吞吐测试使用 golden acceptance length 3.69 而不是 3.41。decode capture sizes 从 6、12 改为 8、16，以匹配 max-num-seqs 2 x (1 + 7)。 - 并发 10/16/24/32 的 max-num-seqs 从 6/10/16/24 改为 7/8/13/20。并发 10 新增 capture size 42（7 x (1 + 5)）。除此之外，这些点的草稿长度、capture 列表、max-num-batched-tokens 和 gpu-memory-utilization，以及并发 48 的所有设置均不变。 - DSpark 草稿精度无变化：Inferact/Kimi-K3.1-Pro-DSpark-preview（revision 6a2ef20a）发布的权重全部为 BF16 且没有 quantization_config，speculative config 指定了草稿且未设置量化，因此 get_draft_quant_config() 返回 None，K3DSparkModel 以 BF16 加载各层，并使用目标模型的 BF16 embedding 和 LM head。其 KV cache 保持 FP8，与目标模型相同。并发 1 的 DCP1 和并发 4 的 7 个草稿 token 都不改变这些类型。 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: seungrokj --- .../kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml | 17 ++++++++++------- inferencex-e2e/configs/amd-master.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 12 ++++++++++++ 3 files changed, 23 insertions(+), 8 deletions(-) diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml index 045d0cfb49..c7c3ea7e5f 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml @@ -102,6 +102,9 @@ override_gpu_kv_c1: agg: args: max-num-seqs: 1 + decode-context-parallel-size: null + dcp-comm-backend: null + cp-kv-cache-interleave-size: null compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[2,4,6,8,10,12,14,16]}' additional-config: '{"decode_backend":"latency_specialized","kda_prefill_backend":"fused"}' speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' @@ -119,8 +122,8 @@ override_dram_c4: agg: args: max-num-seqs: 2 - compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' - speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[8,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: CONC: '4' @@ -131,8 +134,8 @@ override_dram_c10: roles: agg: args: - max-num-seqs: 6 - compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' + max-num-seqs: 7 + compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,42,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: env: @@ -144,7 +147,7 @@ override_dram_c16: roles: agg: args: - max-num-seqs: 10 + max-num-seqs: 8 compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: @@ -157,7 +160,7 @@ override_dram_c24: roles: agg: args: - max-num-seqs: 16 + max-num-seqs: 13 compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}' speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' benchmark: @@ -170,7 +173,7 @@ override_dram_c32: roles: agg: args: - max-num-seqs: 24 + max-num-seqs: 20 max-num-batched-tokens: 10240 gpu-memory-utilization: 0.91 compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}' diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 86c68fcafd..08e27bff23 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -941,7 +941,7 @@ kimik3.1-fp4-mi355x-vllm-agentic-dspark: - dram-utilization: 0.6 search-space: # The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 keeps the KV cache on the GPUs. - - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml } + - { tp: 8, ep: 1, dcp-size: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml } - { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 10, 16, 24, 32, 48], srt-recipe: agentic.yaml } # MiniMax-M3.1-Flash AgentX on MI355X with SGLang: day-zero image from the SGLang diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 4d7d1d548a..f1f208478c 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12449,3 +12449,15 @@ - "Bump to lmsysorg/sglang-rocm:muse-spark-open-mi355-1106." - "Concurrency 6, 10 and 28: TP8/EP8 with GPU-only KV; 28, 36 and 40: TP4/EP4 with a 310 GB-per-rank HiCache DRAM tier." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4881 + +- config-keys: + - kimik3.1-fp4-mi355x-vllm-agentic-dspark + scenario-type: + - agentic-coding + description: + - "The image stays vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336)." + - "Concurrency 1 moves from TP8 x DCP8 to TP8 x DCP1 (master config dcp-size 8 -> 1; the recipe's c1 override drops decode-context-parallel-size, dcp-comm-backend and cp-kv-cache-interleave-size). It keeps GPU-resident KV, 7 draft tokens and the latency_specialized decode backend. Concurrency 4-48 stay TP8 x DCP8 with DRAM offload." + - "Concurrency 4: DSpark draft tokens 5 -> 7, so throughput runs use golden acceptance length 3.69 instead of 3.41. The decode capture sizes change from 6, 12 to 8, 16 to match max-num-seqs 2 x (1 + 7)." + - "max-num-seqs at concurrency 10/16/24/32 changes from 6/10/16/24 to 7/8/13/20. Concurrency 10 adds capture size 42 (7 x (1 + 5)). Otherwise the draft lengths, capture lists, max-num-batched-tokens and gpu-memory-utilization at these points, and every setting at concurrency 48, are unchanged." + - "No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config names the draft and sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache stays FP8, the same as the target's. DCP1 at concurrency 1 and 7 draft tokens at concurrency 4 do not change these types." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4887 From 120d82d6a55248d2127b49d2c376d6ceb7573aaf Mon Sep 17 00:00:00 2001 From: Alec Ibarra <93070681+adibarra@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:48:02 -0600 Subject: [PATCH 040/111] =?UTF-8?q?Speed=20up=20matrix=20tooling,=20AgentX?= =?UTF-8?q?=20scoring=20and=20tests;=20share=20the=20dataset=20cache=20wit?= =?UTF-8?q?h=20single-node=20AgentX=20/=20=E5=8A=A0=E9=80=9F=E7=9F=A9?= =?UTF-8?q?=E9=98=B5=E5=B7=A5=E5=85=B7=E3=80=81AgentX=20=E8=AF=84=E5=88=86?= =?UTF-8?q?=E4=B8=8E=E6=B5=8B=E8=AF=95=EF=BC=9B=E5=8D=95=E8=8A=82=E7=82=B9?= =?UTF-8?q?=20AgentX=20=E5=85=B1=E4=BA=AB=E6=95=B0=E6=8D=AE=E9=9B=86?= =?UTF-8?q?=E7=BC=93=E5=AD=98=20(#4918)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(matrix): reuse head points for config keys a base leaves unchanged `python -m infx.matrix.changed --base REF` ran REF's generator for every config key, although most keys are the same on both sides. When REF and the working tree hold the same generator code, recipes and runner config, a key whose master entry is unchanged now takes the working tree's points as its base points. Git decides "the same": no tracked difference from REF under the snapshot paths, and every file there that Git does not ignore is a tracked regular file whose content Git checks (no untracked file, no skip-worktree or assume-unchanged bit, no symlink). A key whose head rows have no fingerprint still runs REF's generator, because its messages name the revision's own paths, and so does a key that is also a pattern. Any other tree runs REF's generator for every key, as before. INFX_REUSE_UNCHANGED=0 turns the reuse off. One changed key against the commit before it, 4 CPUs: 8.0 s -> 1.85 s when the tree differs from REF in master configs only. A tree that also changes a recipe, a doc or code under the snapshot paths takes the previous path. * perf(srt-slurm): compose a single-node fragment once per planning run The planner selected the recipe of every single-node point by composing its fragment, expanding all the variants and binding each of them. select_recipe now composes and expands a fragment once per process for the planner, keyed by the text of every file compose_recipe reads, and binds only the variants whose CONC can be the point's. Anything but exactly one match, and any error, takes the previous path, so the selected recipe and every message are the same. The bound-key check of a fragment runs once per fragment text, and a zip selector expands only the variant it names. 4 CPUs, a780b7667: plan of every key 2.54 s -> 1.62 s, full generate 2.38 s -> 1.46 s, Klaud plan 2.30 s -> 1.45 s, `infx generate` of the srt-slurm keys 6.53 s -> 6.17 s. * test: schedule the suite with worksteal and skip the Klaud read backoff With xdist's default scheduling one worker ran alone for 20 s after the other three had finished. `--dist worksteal` hands its queued tests to idle workers. The Klaud tests also waited out the real backoff of retried GitHub reads, 5 s in total; they now stub that sleep, and the tests that check the delays still record them. CI's Tests command on 4 CPUs: 42.1 s -> 26.9 s. * feat(launch): share the replay dataset cache with single-node AgentX jobs A single-node AgentX job rebuilt the replay dataset before every benchmark: 2.0 to 3.7 minutes in five sampled jobs, while the server held its GPUs. Multi-node AgentX jobs restore the same dataset from the cluster's aiperf-cache volume in about 4 s, because the launcher mounts the volume and sets AIPERF_DATASET_MMAP_CACHE_DIR. The launcher now does the same for a single-node AgentX benchmark on a cluster whose srt-slurm profile lists aiperf-cache under volume-mounts or agentic-volume-mounts (h200-dgxc, gb300-nv, b200-nscale, gb200-nv). Eval-only and fixed-sequence jobs do not change, and neither do clusters without the volume. A single-node AgentX fragment may no longer set the variable itself. * perf(results): parse only the replayed traces when scoring an AgentX point Scoring a point parsed every trace of the public dataset, 1.85 GB of JSON lines, to look up the expected output lengths of the traces the run replayed: 45 of 393 in a sampled 3,600 s point. iter_trace_blobs now takes the ids of the replayed traces and skips a *.jsonl line unparsed when the line is an object written without escapes and none of its "id" keys holds a wanted id. Every line that can hold a wanted trace is parsed as before, so the statistics are the same. One behavior changes: a malformed trace that the run did not replay no longer fails the point. Before, a trace whose turns could not be read raised while the index of all traces was built. Scoring the sampled point: 11.2 s -> 3.75 s, peak RSS 1.9 GB -> 1.3 GB. --- .github/workflows/ci.yml | 2 +- AGENTS.md | 2 +- .../multi_node/srt-slurm-recipes/RECIPES.md | 2 +- inferencex-e2e/configs/CONFIGS.md | 3 +- .../infx/launch/drivers/srt/__init__.py | 6 +- .../infx/launch/drivers/srt/config.py | 20 ++ inferencex-e2e/infx/matrix/changed.py | 157 +++++++++++--- .../infx/results/agentic/artifacts.py | 39 +++- .../results/agentic/process_agentic_result.py | 3 +- .../infx/results/agentic/request_metrics.py | 8 + inferencex-e2e/infx/srt_slurm/generate.py | 2 +- inferencex-e2e/infx/srt_slurm/single_node.py | 119 +++++++++-- inferencex-e2e/infx/srt_slurm/variants.py | 20 +- inferencex-e2e/infx/srt_slurm/workload.py | 34 ++- inferencex-e2e/infx/tests/klaud/conftest.py | 6 + .../tests/klaud/test_klaud_reliability.py | 2 + .../infx/tests/launch/test_srt_driver.py | 27 +++ .../infx/tests/matrix/test_changed.py | 198 ++++++++++++++++++ .../agentic/test_process_agentic_result.py | 62 +++++- .../tests/srt_slurm/test_srt_single_node.py | 80 +++++++ .../infx/tests/srt_slurm/test_variants.py | 5 + 21 files changed, 715 insertions(+), 82 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 69bff7d3c7..f73db90ca6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -89,4 +89,4 @@ jobs: if: matrix.check == 'Tests' run: | python -c "import torch; assert torch.version.cuda is None and torch.version.hip is None" - python -m pytest -c inferencex-e2e/pyproject.toml inferencex-e2e/infx/tests/ inferencex-e2e/utils/ inferencex-e2e/runners/ collectivex/tests/ operatorx/tests/ --ignore=inferencex-e2e/utils/srt-slurm -n 4 + python -m pytest -c inferencex-e2e/pyproject.toml inferencex-e2e/infx/tests/ inferencex-e2e/utils/ inferencex-e2e/runners/ collectivex/tests/ operatorx/tests/ --ignore=inferencex-e2e/utils/srt-slurm -n 4 --dist worksteal diff --git a/AGENTS.md b/AGENTS.md index a02a1f1563..2926428c50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,7 +76,7 @@ Run from `inferencex-e2e/`: uv sync --locked --all-extras --group test uvx --exclude-newer PT12H ruff@latest check --fix infx uvx --exclude-newer PT12H ruff@latest format infx -uv run --locked --all-extras --group test python -m pytest -c pyproject.toml infx/tests/ -n 4 +uv run --locked --all-extras --group test python -m pytest -c pyproject.toml infx/tests/ -n 4 --dist worksteal ``` Run the focused test path first. Use `--extra workflows` for GitHub integrations and `--extra results` for result tooling. Dependency changes update `pyproject.toml` and `uv.lock` via `uv add` or `uv lock --upgrade-package NAME`. Workflow changes run the [Zizmor CI command](.github/workflows/zizmor.yml) with online checks; do not weaken its policy. Schema and generator checks do not prove serving behavior: run the exact config, inspect its generated fields, then collect smoke, sweep, and eval evidence. diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md index d3c19b21e3..44cb5799a6 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md @@ -89,7 +89,7 @@ Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments 1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). Every multi-node benchmark point on a cluster with a GPU exporter also gets a telemetry block, with the cluster's CPU exporter where it declares one, unless its recipe runs etcd and NATS on a dedicated node: [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml) where the cluster sets `srt-slurm.power-exporter-port`, else [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml). The binder sets `telemetry.required` only for a row with `power: true`. The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. A native single-node point on a cluster that declares `srt-slurm.cpu-power-exporter`, and a `cpu-power: true` row, also gets `telemetry.cpu_power_exporter` with that source. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches. 2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning. -3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:`, `model.container: `, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power, or any GB200 or GB300 AgentX point the launcher measures by default) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments. +3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:`, `model.container: `, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. The launcher also sets `AIPERF_DATASET_MMAP_CACHE_DIR` for a single-node AgentX benchmark when it submits, on a cluster whose `volume-mounts` or `agentic-volume-mounts` list `aiperf-cache`, so a fragment does not set it. Multi-node AgentX recipes also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power, or any GB200 or GB300 AgentX point the launcher measures by default) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments. 4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/