Repository navigation
Merge InferenceX main / 合并 InferenceX main - #6
Merged
Merged
Conversation
…ench): 新增 GB300 上的 GLM-5.3-Flash FP8 与 NVFP4 AgentX (SemiAnalysisAI#4715) * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 GB300 AgentX lane * chore(bench): register GLM-5.3-Flash FP8 GB300 sweep * fix(ci): preserve canary gates after skipped reuse checks * fix(bench): use canonical Flash tokenizer ID for AgentX * feat(bench): add GLM-5.3-Flash FP8 and NVFP4 AgentX on GB300 Single-node SGLang v0.5.22 TP4 MTP 5/1/6 lanes for FP8 and NVFP4 using the SGLang cookbook GB300 recipe: GPU-resident KV at low concurrency and an auto-sized HiCache host tier (host memory fraction 0.90) at high concurrency. * feat(bench): move GLM-5.3-Flash GB300 lanes to SGLang v0.5.23 Pin lmsysorg/sglang:v0.5.23-cu130 for the FP8 and NVFP4 GB300 lanes and declare the expected SGLang version and FP8 checkpoint revision in the recipe identities. * revert GLM-5.3-Flash GB300 lanes to SGLang v0.5.22 Return both GB300 lanes to lmsysorg/sglang:v0.5.22-cu130 and declare SGLang 0.5.22 in the recipe identities. * fix(bench): record measured GB300 HiCache DRAM for GLM-5.3-Flash Set dram-utilization to 0.7495 (723 GB per node) on both GB300 lanes, covering the 708.7-722.9 GB per node the auto-sized HiCache pools allocated in run 42026173061. --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
… FP8 与 NVFP4 B300 SGLang AgentX (SemiAnalysisAI#4710) * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * feat: add GLM-5.3-Flash FP8 B300 AgentX recipe * perf: register GLM-5.3-Flash FP8 B300 sweep * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * fix: use SGLang v0.5.22 decode CUDA graph flag * feat(agentx): add GLM-5.3-Flash FP8 and NVFP4 B300 SGLang MTP lanes Serve both precisions with the SGLang cookbook's B300 Low Latency flags: GPU-resident KV at concurrency up to 48, and at concurrency 64 and 96 HiCache with auto-sized host pools (fraction 0.76, within the 3 TB node cap) and max-running-requests 96, which SGLang otherwise resets to 48 under EAGLE. * fix(agentx): record measured HiCache DRAM for GLM-5.3-Flash B300 Run 42024862281 allocated up to 2802.2 GB (FP8) and 2813.6 GB (NVFP4) of HiCache host pools per node; dram-utilization 0.935 and 0.939 record 2804 and 2816 GB. --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
…ash FP8 H100 SGLang AgentX（含 HiCache） (SemiAnalysisAI#4717) * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 H100 AgentX lane * chore(bench): register H100 GLM-5.3-Flash full sweep * chore(debug): probe H100 checkpoint inventory read-only * fix(ci): preserve canary gates after skipped reuse checks * chore(debug): remove completed H100 inventory probe * fix(bench): stage pinned GLM-5.3-Flash checkpoint for H100 * fix(bench): verify pinned H100 cache before offline serving * fix(bench): pin H100 server tokenizer to cached checkpoint * fix(bench): retain canonical AgentX tokenizer resolution on H100 * fix(bench): clear offline flags for the H100 AgentX client * fix(bench): test eager prefill compatibility on H100 * feat(bench): consolidate GLM-5.3-Flash FP8 H100 AgentX onto the cookbook recipe Rebuild the H100 lane on main with the SGLang cookbook H100 low-latency FP8 recipe: GPU-resident KV at concurrency 1-2 and the cookbook HiCache L1+L2 host tier (32 GB of host KV per rank) at concurrency 4-16. Keep prefill CUDA graphs disabled for the H100 long-context replay fault and stage the pinned checkpoint into the shared HF cache. * fix(bench): record the H100 HiCache host allocation as its DRAM share * fix(bench): drop the no-op trust-remote-code from the H100 recipe * fix(bench): size H100 HiCache DRAM from the KV/Mamba split and move points to c4-c12 --hicache-size 32 is shared by the KV and Mamba host pools in proportion to their device bytes, so host KV is about 22 GB per rank; with the DSA indexer pool each rank allocates about 32.7 GB. * fix(bench): drop H100 HiCache c12 and restate the HiCache host allocation * chore(bench): drop H100 GLM-5.3-Flash checkpoint staging The pinned checkpoint is already in the shared H100 Hugging Face cache, which the server reads offline through its pinned tokenizer path, as the MI300X and MI325X lanes on main do. * chore: refresh PR SemiAnalysisAI#4717 for sweep reuse [skip-sweep] --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
…iAnalysisAI#4833) Run the Kimi-K3.1-Pro B300 vLLM recipe's low-latency decode (decode_backend=latency_specialized) at c1-c4, run c1-c2 at plain TP8 instead of TP8 x DCP8, match the SGLang recipe's DSpark draft depths, size each decode graph cap to max-num-seqs x (1 + drafts), and raise the engine-ready timeout and health check to 7200 s. Co-authored-by: zjy0516 <zhujiangyun@inferact.ai>
… 在 1、2、4、8 张 B200 上扫描 H3 延迟 (SemiAnalysisAI#4852) * feat: sweep H3 video on 1, 2, 4 and 8 B200 GPUs * chore: link B200 GPU-count changelog entry to SemiAnalysisAI#4852
…2、4、8 张 H100 上扫描 H3 延迟 (SemiAnalysisAI#4857) * feat: sweep H3 video on 2, 4 and 8 H100 GPUs * chore: link H100 GPU-count changelog entry to SemiAnalysisAI#4857
* feat(kimik3.1): add day-0 vLLM AgentX on GB200 Aggregated TP8 x PP2 x DCP8 and disaggregated 1P1D TP16 Kimi K3.1 Pro points on gb200-nv with the K3.1 DSpark draft and Simple CPU KV offload, plus the lane's shared run root and AgentX power rule. * chore(kimik3.1): link the GB200 vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Remove settings that no active recipe uses and vLLM's K3.1 recipe does not document: the DCP comm backend, prompt-token details, and the FlashInfer all-reduce, execute-timeout, UCX and AIPerf failure-threshold env vars. * fix(kimik3.1): pin Dynamo #18124 and use the MXFP4 indexer on GB200 vLLM PD Published Dynamo wheels cannot load the kimi_k31 tokenizer, so the PD frontend 404s. Pin ai-dynamo/dynamo#18124, which pulls the K3.1 frontend crates, and record it as the router version. The PD roles also use the MXFP4 indexer cache, which the week-1 rules allow and vLLM's K3.1 DSA path supports with NIXL. * fix(kimik3.1): front the GB200 vLLM PD row with the vLLM Router No published Dynamo frontend loads the kimi_k31 tokenizer and the K3.1 Dynamo branch does not build, so the 1P1D row runs direct vLLM workers behind vllm-router 0.1.14 with the same engine arguments, NIXL transfer and MXFP4 indexer. The GB200 Kimi-K3.1 AgentX power rule now also covers vllm. * fix(kimik3.1): label the GB200 vLLM rows as vllm Neither GB200 row runs Dynamo: the aggregated row uses the direct vLLM frontend and the PD row the vLLM Router. Name both keys and their framework vllm, drop the Dynamo-only recipe settings, and scope the Kimi-K3.1 AgentX power rule to vllm. * fix(kimik3.1): keep GB200 vLLM PD worker connections alive for the router vllm-router reuses pooled connections that Uvicorn closes after its 5 s default keep-alive, and with retries disabled the reset aborts the request. Hold PD worker connections for 600 s, as main's GB200 vLLM PD recipe does. * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-gb200-vllm): run disagg decode with expert parallelism Decode at TP16/EP1 holds 131.5 GiB/GPU of weights and leaves 12-19 GiB of KV, less than the 18.17 GiB one 1M-token request needs at c32 and c48. EP16, as on prefill, cuts decode weights to about 103.5 GiB/GPU.
…增基于 cookbook 配方的 GLM-5.3-Flash FP8 MI355X SGLang AgentX (SemiAnalysisAI#4720) * feat(speedbench): collect GLM-5.3-Flash natural MTP acceptance * fix(speedbench): use vLLM B300 for GLM-5.3-Flash collection * fix(speedbench): preserve multi-turn rows and drain GPU workers * feat(golden-al): commit measured GLM-5.3-Flash MTP curve * fix(ci): allow authorized sweeps on stacked pull requests * fix(bench): preserve golden acceptance precision in native overrides * feat(bench): add native GLM-5.3-Flash FP8 MI355X lane * chore(bench): record native Flash MI355X sweep provenance * fix(ci): preserve canary gates after skipped reuse checks * fix(bench): disable integer QuickReduce for Flash MI355X * fix(bench): verify pinned Flash cache before offline serving * fix(bench): pin Flash serving tokenizer to staged snapshot * fix(bench): restore canonical Flash client tokenizer lookup * fix(bench): clear Flash client offline flags for tokenizer warmup * fix(bench): use upstream Flash hybrid attention fix on MI355X * feat(bench): consolidate GLM-5.3-Flash FP8 MI355X onto the SGLang cookbook recipe Rebuild the MI355X FP8 lane on current main with the cookbook MI355X FP8 Low Latency recipe: TP4/EP4, HBM-only KV at conc 1-32 and the cookbook HiCache L1 + L2 tier at conc 40, 48 and 64. * fix(bench): record measured GLM-5.3-Flash MI355X HiCache DRAM and page size The c40/c48 HiCache servers allocate 187.33 GB of host memory per rank (749.33 GB per TP4 server); record 751 GB. SGLang v0.5.22 forces page size 256 for DSA models, so state it. * docs(bench): state the measured MI355X HiCache host pools per rank * chore: refresh PR SemiAnalysisAI#4720 for sweep reuse [skip-sweep] --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
…Day-0 支持 (SemiAnalysisAI#4806) * feat(kimik3.1): add Kimi K3.1 Pro vLLM AgentX on MI355X Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification, following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs. GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60. * chore(kimik3.1): link the MI355X vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Write the per-concurrency points as plain override variants instead of zip overrides. * fix(kimik3.1): leave GPU headroom for pinned scratch on MI355X The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38 needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X recipe did on the same nodes. * perf(kimik3.1): match the apex-1104 sweep's points and batching The MI355X AgentX lane and the reference apex-1104 sweep differed in the points they ran and in batching, so a comparison between them could not attribute anything to the engine build. Align the parts that shape the workload: seven points instead of ten (drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs below concurrency as that sweep sets it, max-num-batched-tokens 16384 except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of host DRAM for the Simple CPU offload. Keep this lane's own precision and speculative settings: 5 DSpark tokens with block rejection at every point, the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state, BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and no FP8 indexer override or KDA input-projection quantization. Those are deliberate here, and copying them would change what the runs measure -- synthetic acceptance in particular reports a modelled throughput and makes the output nondeterministic by construction. The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% -- the recipe's cpu_bytes_to_use_per_rank wins. Say so instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(kimik3.1): resolve the MI355X DRAM budget to the recipe's 2577 GB The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to 2577 GB, within the 3 TB per-server limit. Align comments with the settings they describe and correct the changelog: throughput points still run synthetic acceptance at the golden 3.08. * perf(kimik3.1): use 7 DSpark tokens and the specialized decode at conc 1 The conc-1 latency point runs the reference sweep's latency_specialized decode backend with 7 draft tokens; the throughput points stay at 5. K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only and breakable graphs are the only thing that would add the prefill sizes. Turn them off and leave scratch reclaim on, as the image sets it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): prefill with the AITER FA MLA backend at conc 48 only The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves conc 1-32 on the default. Move it out of base into that point's override. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): set breakable cudagraphs and scratch reclaim per point The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the conc-1 values into that point's override and leave the base on the sweep's throughput setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(kimik3.1): keep scratch reclaim off on the MI355X DRAM points On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at conc 4-48 and no async scratch limit there; conc 1 keeps its own settings. * perf(kimik3.1): trim the conc-32 and conc-48 points on MI355X Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no successful request. Cut the per-step working set at the two largest points: max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens, with the capture lists rebuilt from those. The other five points keep the reference apex-1104 sweep's settings. Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill backend at its default everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): keep scratch allocated and step the draft count down MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release blocks until the running dispatches finish, which is how the conc-16 and conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a reclaim that no longer happens. Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB that dram-utilization 0.6 resolves to. Describe the points on their own terms rather than against another sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-mi355x-vllm): lower gpu-memory-utilization to 0.86 at conc 16-32 With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest. * fix * fix * perf(kimik3.1): refresh the day-0 image and map the offload pool with huge pages THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the 224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55 million of them to register and keep mapped, against 107 thousand 2 MiB pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): sync Kimi-K3.1-Pro MI355X vLLM block with the diff [skip-sweep] - Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2). - TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to). - Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48. - Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens. - Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts. - K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE). - Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: refresh PR SemiAnalysisAI#4806 for sweep reuse [skip-sweep] --------- Co-authored-by: billishyahao <bill.he@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: seungrokj <seungrok.jung@amd.com>
… image to v0.33.0 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 MI325X vLLM AgentX DSpark 镜像更新至 v0.33.0 (SemiAnalysisAI#4794) * chore(dsv41flash): bump MI325X vLLM AgentX DSpark image to v0.33.0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(dsv41flash): add MI325X vLLM v0.33.0 perf-changelog entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…mage to nightly-00b6c4c / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 H100 vLLM AgentX DSpark 镜像更新至 nightly-00b6c4c (SemiAnalysisAI#4569) * chore(dsv41flash): update H100 vLLM AgentX image to nightly-00b6c4c Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(perf-changelog): add DSv4.1 Flash H100 vLLM AgentX image refresh Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
…image to nightly-00b6c4c5 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 GB300 vLLM AgentX DSpark 镜像更新至 nightly-00b6c4c5 (SemiAnalysisAI#4558) * config: update GB300 DSV4.1 Flash vLLM AgentX DSpark image to nightly-00b6c4c5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf-changelog: add GB300 DSV4.1 Flash vLLM AgentX DSpark nightly-00b6c4c5 entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
…iAnalysisAI#4839) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… 在 1、2、4、8 张 H200 上扫描 H3 延迟 (SemiAnalysisAI#4856) * feat: sweep H3 video on 1, 2, 4 and 8 H200 GPUs * chore: link H200 GPU-count changelog entry to SemiAnalysisAI#4856
…X] 在 1、2、4、8 张 MI355X 上扫描 H3 延迟 (SemiAnalysisAI#4818) * feat: run native H3 video sweeps from YAML recipes * feat: verify GPU ownership and seal H3 execution evidence * feat: define reproducible H3 workload sets * feat: measure H3 requests and validate generated media * refactor: delete unused OwnedProcess.owns helper * refactor: delete unused H3 bundle JSON digest helper * test: restore shared launch and workflow test cases * refactor: fold H3 spec digest and case helpers * test: table-drive H3 spec rejection cases * chore: compact H3 spec YAML and drop stale references * fix: use warmup case in GPU report slots * refactor: drop unreachable GPU job guards * test: drop dead parametrize branches in measurement tests * refactor: share no-follow open and canonical JSON helpers * test: inline single-case video parametrize and share reseal fixtures * refactor: share deadline budget in GPU probes * refactor: reuse runner slots and hashing in compare * test: drop dead parametrize branches in GPU tests * refactor: dedupe evaluator encoding, status, and quantile rules * refactor: drop duplicated video digest, srun, schema and env helpers * refactor: share video srt recipe defaults like agentic * docs: link video pins and timeouts to their sources * refactor: move offline compare and report tools out of measurement core * refactor: move offline GPU job tools out of native H3 sweep * refactor: move manual srt-yaml commands out of native H3 sweep * style: format native H3 evaluator modules with ruff * refactor: expose canonical_json_bytes as the one H3 hash encoding * fix: check the video readiness port and drop the empty prepare description * refactor: drop citations and per-engine notes from H3 specs * test: drop the hand-built fairness hash test * refactor: validate H3 v0.1 policy once in the spec loader by schema version * ci: lint and format all of experimental/video-generation and drop measurement-only deps * revert: keep run comparison and PSNR in the measurement core * refactor: move SGLang server timing out of the measurement core * chore: drop the unreferenced smoke plan and the nonexistent mvp extra hint * style: ruff format the measurement core * refactor: share the native video file contract from infx.bench.video * refactor: hash run bundles with the spec's canonical JSON * refactor: name each rejected field of the sealed v0.1 server launch * refactor: trust loader-validated plans and drop the redundant run_plan model_revision * refactor: derive engine encoding requests and media checks from the plan's encoding * refactor: keep native video checks at the binder, executor and ingest boundaries * chore: drop unconsumed prose and constant fields from run data * fix: keep a bounded redacted HTTP error excerpt and refuse every paid MiniMax host * refactor: default the single-choice reference dtype and attention backend * refactor: record native video job status as JSON * fix: count malformed power samples in the video bundle instead of skipping them * fix: drop unread constant bundle fields and validate each power sample's GPU coverage * refactor: resolve the video harness at one path beside the project * fix: hash the measurement window run with the shared canonical JSON * refactor: seal the image digest for every native video plan and check the launch once * fix: keep the video workload specs in nested matrix snapshots * fix: report the native video bundle outcome as its CI phase * fix: count video sweep jobs in run stats after they finish * fix: drop the static protocol source pin so run configuration names only the image revision * refactor: rename generic H3 spec and workflow names to video * refactor: rename generic H3 measurement-core naming to video * refactor: rename generic H3 native-sweep naming to video * fix: record approved head and video matrix in trusted sweeps * fix: validate trusted video sweeps with the dispatching workflow's validator * feat: run H3 on SGLang v0.5.23 for B200 and H100 (SemiAnalysisAI#4776) * fix: record engine parallelism in native video spec.json (SemiAnalysisAI#4777) * chore: fold the v0.5.23 changelog entry into the H3 sweep entry * chore: drop the H200 vLLM-Omni cell from the H3 sweep * chore: run H3 H200 on SGLang v0.5.23 * chore: drop the H200 SGLang cell from the H3 sweep * refactor: move the video harness to top-level video-generation * refactor: name the 20-prompt H3 spec by its schema version * docs: drop the release label from the runner's policy note * fix: let vgbench run record the server's numerical identity * fix: check reference case IDs before selecting all cases * docs: drop the release label from the H3 docs and changelog entry * fix: accept the newest video bundle per deployment across rerun attempts * fix: retry the video results upload like other result uploads * fix: stop serving-mode video runs promptly on interrupt * fix: name ROCm video GPUs by board serial * feat: add MiniMax-H3 MI355X SGLang video sweep * feat: serve MI355X H3 video from the shared HF cache * docs: link the MI355X H3 changelog entry to its PR * fix: select AITER attention in the MI355X H3 recipe * feat: sweep H3 video latency across 1, 2, 4 and 8 GPUs * fix: keep a single video bundle in its named directory [skip-sweep] * chore: link GPU-count changelog entry to SemiAnalysisAI#4818 [skip-sweep] * ci: raise video job limit to 150 minutes for 1-GPU points * refactor: share single-node srt steps between points and video [skip-sweep] * feat: sweep B200 H3 video across 1, 2, 4 and 8 GPUs * chore: move B200 GPU-count points to SemiAnalysisAI#4852 * chore: narrow the GPU-count sweep to MI355X
…-Flash-Next FP4 GB300 SGLang AgentX 更新 (SemiAnalysisAI#4787) * [NV] Qwen3.8-Flash-Next FP4 GB300 SGLang AgentX update Signed-off-by: hshrivastava-droid <hshrivastava@nvidia.com> * Update perf-changelog pr-link Signed-off-by: hshrivastava-droid <hshrivastava@nvidia.com> * Update perf-changelog description Signed-off-by: hshrivastava-droid <hshrivastava@nvidia.com> --------- Signed-off-by: hshrivastava-droid <hshrivastava@nvidia.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
… image to v0.33.0 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 MI300X vLLM AgentX DSpark 镜像更新至 v0.33.0 (SemiAnalysisAI#4793) * chore(dsv41flash): update MI300X vLLM AgentX image to v0.33.0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(dsv41flash): add MI300X vLLM AgentX v0.33.0 perf-changelog entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
…k image to nightly-dev-cu13-20261027-5884e4fe / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 GB200 SGLang AgentX DSpark 镜像更新至 nightly-dev-cu13-20261027-5884e4fe (SemiAnalysisAI#4591) * perf(dsv41flash): update GB200 SGLang AgentX DSpark image to nightly-dev-cu13-20261027-5884e4fe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(dsv41flash): add perf-changelog entry for GB200 SGLang AgentX DSpark image update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
… image to nightly-dev-cu13-20261028-586355a8 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 H200 SGLang AgentX DSpark 镜像更新至 nightly-dev-cu13-20261028-586355a8 (SemiAnalysisAI#4597) * chore(dsv41flash): update H200 SGLang AgentX DSpark image to nightly-dev-cu13-20261028-586355a8 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(dsv41flash): add perf-changelog entry for H200 SGLang AgentX DSpark image update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
… image to nightly-dev-cu13-20261029-c10e0e17 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 B200 SGLang AgentX DSpark 镜像更新至 nightly-dev-cu13-20261029-c10e0e17 (SemiAnalysisAI#4635) * chore(configs): update DeepSeek-V4.1-Flash FP4 B200 SGLang AgentX image to nightly-dev-cu13-20261029-c10e0e17 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(perf-changelog): record DeepSeek-V4.1-Flash FP4 B200 SGLang AgentX image update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
…ly-dev-cu13-20261027-5884e4fe / [Klaud Cold] 将 GLM-5.2 FP4 B200 SGLang AgentX MTP 镜像更新至 nightly-dev-cu13-20261027-5884e4fe (SemiAnalysisAI#4573) * chore(glm5.2): update FP4 B200 SGLang AgentX MTP image to nightly-dev-cu13-20261027-5884e4fe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(glm5.2): add perf-changelog entry for FP4 B200 SGLang AgentX MTP image update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
…Flash-Next B200 FP4 SGLang AgentX 更新 (SemiAnalysisAI#4850) * Qwen3.8-Flash-Next B200 FP4 SGLang AgentX Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Update perf-changelog PR link Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Remove Qwen3.8-Flash-Next B200 FP4 SGLang AgentX concurrency 1 and 4 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Ankur Singh <ankusingh@nvidia.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…#4846) * [AMD][MI35X] 1105 Kimi K3.1 Pro sglang dflash agentic - Image: lmsysorg/sglang-rocm:dev-kimi-k31-pro-mi35x-1105 (day-0 AMD SGLang with #48437, #48439 and #48440; sgl-amd-mono e52803f9d2). - Concurrency 1 and 4: max-mamba-cache-size 60 -> 150. * Link perf-changelog entry to SemiAnalysisAI#4846 * Row-sharded MoE front and stream at concurrency 16, 32 and 48 * Keep only the reason in the Mamba cache comment * Drop the row-sharded MoE front and stream at concurrency 16, 32 and 48 * Concurrency 48: 32k prefill chunks every 8 decode steps, row-sharded residual stream Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Concurrency 48: back to 16k prefill chunks every 16 decode steps Both c48 attempts with 32k chunks every 8 decode steps were killed mid-profile (run 42161676877, nodes mia1-p01-g16 and mia1-p01-g09) with no traceback. Keep the row-sharded residual stream, which costs no memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * New image; concurrency 48 and 56: 5-token verify, 32k chunks every 8 decode steps lmsysorg/sglang:dev-kimi-k31-pro-mi35x (sha256:346a6d83) carries SGLang #48427 at 64507491d9: DCP shares the TP communicators (about 9 GiB a GPU back), the latent-MoE tail column-splits up to 384 rows, the radix-4 router, and ROCm host pools sized against the amdgpu pinned-memory limit; sgl-amd-mono 25723f1e46. Concurrency 48, and a new 56: DFLASH verifies the current token plus 4 drafts (golden AL 3.25), 32k prefill chunks every 8 decode steps, the MoE front and residual stream on row shards in prefill, and fused draft ring locations. The c48 kills of run 42161676877 were on the 1105 image; on this code both points ran 3600 s on 1x8 MI355X (simulated acceptance): c48 19,233 tok/s/GPU at memory fraction 0.826, c56 19,507 at 0.85. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Concurrency 8 to 56: Mamba host ratio 12 -> 10 The new image fits about 20% more KV and Kimi state slots on each GPU, and the HiCache host pools follow them: about 196 GB pinned per GPU at concurrency 56 and more at 32, against 1511 GiB of GTT the node's 8 ranks share. Concurrency 32 failed its hipHostMalloc of a 144.61 GB state pool (run 42322137350, mia1-p01-g09). Ten host states per device state leave 170-185 GB a GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com> Co-authored-by: Kevin Mi <mikevin920@yahoo.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…bine / CollectiveX：在 DeepEP V2 与 UCCL-EP 的 combine 中计入 vLLM 的 finalize 拷贝 (SemiAnalysisAI#4882) * fix(collectivex): charge vLLM's finalize copy in deepep-v2 and uccl-ep normal combine vLLM v0.32.0's deepep_v2 and deepep_ht finalize copy the combined rows into the MoE layer's output; mori and flashinfer-ep already time that copy, these two returned the buffer. Rows record library_config.finalize_copy. Docs: FlashInfer coverage on H100/H200, DeepEP V2's venv torch/NCCL. vLLM v0.32.0 的 deepep_v2 与 deepep_ht finalize 会把合并结果拷贝到 MoE 层输出；mori 和 flashinfer-ep 已计入该拷贝，这两个后端此前直接返回缓冲区。结果在 library_config.finalize_copy 中记录。文档：补充 FlashInfer 在 H100/H200 的覆盖以及 DeepEP V2 venv 的 torch/NCCL。 * docs(collectivex): record the measured cost of the finalize copy 文档：记录 finalize 拷贝的实测开销。 * docs(collectivex): add the perf-changelog entries for the finalize copy 为 finalize 拷贝添加 perf-changelog 条目。
…Infer EP8 on B200 / CollectiveX：DeepEP V2 改用镜像自带的 torch 与 NCCL 构建；在 B200 上运行 FlashInfer EP8 (SemiAnalysisAI#4883) * feat(collectivex): build DeepEP V2 on the image's torch and NCCL; run flashinfer-ep EP8 on b200 DeepEP V2 的构建改用镜像自带的 torch 与 NCCL；在 b200 上运行 flashinfer-ep EP8。 * docs(collectivex): add the perf-changelog entries 添加 perf-changelog 条目。
…ideoGenX] 将 sweep 视频与 Diffusers 参考视频对比打分 (SemiAnalysisAI#4659) * feat: verify GPU ownership and seal H3 execution evidence * feat: define reproducible H3 workload sets * feat: measure H3 requests and validate generated media * refactor: delete unused OwnedProcess.owns helper * refactor: fold H3 spec digest and case helpers * test: table-drive H3 spec rejection cases * chore: compact H3 spec YAML and drop stale references * fix: use warmup case in GPU report slots * refactor: drop unreachable GPU job guards * test: drop dead parametrize branches in measurement tests * refactor: share no-follow open and canonical JSON helpers * refactor: share deadline budget in GPU probes * refactor: reuse runner slots and hashing in compare * test: drop dead parametrize branches in GPU tests * refactor: dedupe evaluator encoding, status, and quantile rules * refactor: move offline compare and report tools out of measurement core * chore: drop parked offline GPU tools superseded by native sweeps * feat: add public H3 reference descriptor * feat: score h3-results clips against public references * ci: score H3 sweep artifacts against the reference release * docs: document H3 reference PSNR scoring * fix: score each deployment's newest bundle and survive unreadable ones
… 提交全流程指南 (SemiAnalysisAI#4885) * docs: add an end-to-end H3 video submission guide * docs: use the standard change and render commands in the video guide * docs: link the render step instead of restating it
…ecord why (SemiAnalysisAI#4894) H100's EFA pool segfaults every rank at EP16 low-latency setup (IBGDA has no EFA transport), so the cell could only ever show as pending. Methodology also records that H100 EP16 normal prefill does not finish inside the per-case guard. 文档：H100 的 DeepEP 低延迟仅注册 EP8，并记录原因。H100 的 EFA 集群在 EP16 低延迟初始化时所有 rank 都会段错误（IBGDA 没有 EFA 传输），该单元只能永远显示为待测。方法学同时记录 H100 EP16 normal prefill 无法在单用例时限内完成。 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…hInfer EP8 (SemiAnalysisAI#4895) * feat(collectivex): run flashinfer-ep EP8 on b300 Hand runs from cx-probe-image-stack on 2026-11-07: BF16 and FP8, decode and prefill, oracle green; within about 1% of b200. 功能：在 b300 上运行 flashinfer-ep EP8。2026-11-07 手动运行，BF16 与 FP8 的 decode 与 prefill 均通过 oracle，与 b200 相差约 1%。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(collectivex): add the perf-changelog entry 文档：添加 perf-changelog 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…、启动与 CI 流程并清理无用代码 (SemiAnalysisAI#4953) * fix(matrix): reject snake_case keys in master configs The config models accepted field names as well as their kebab-case aliases, while the generator reads only kebab-case keys from the raw YAML. A key such as spec_decoding validated and was then ignored. * fix(speedbench): record N/A when the benchmark client fails A collector whose client exited non-zero still computed acceptance length from the counters the partial pass left behind, and the workflow publishes that value into a golden curve. * fix(ci): collect results for agentic-only sweeps without a canary collect-results listed the agentic sweep jobs in needs but not in its condition, so a non-canary sweep with only agentic rows skipped result collection, comparison and the success rate. * fix(merge): post the reuse command only once the branch is ready to push The helper posted /reuse-sweep-run before fetching and merging main. A merge that then aborted left a standing authorization on the PR, so later pushes skipped their sweep. * fix(reuse): require write access to authorize sweep reuse A reuse command was trusted by the comment's author_association, which admits org members and collaborators without write access. Staging and sign-off already check the repository permission, so reuse now applies the same rule to the comment author. * fix(operatorx): fail planning when the registry does not answer for an image A digest probe that failed for any reason was read as a pruned tag and the shard was moved to the backend's own image. Only a registry 404 now takes that fallback. * fix(launch): keep following a job while squeue is failing follow_log and wait_for_log read a squeue that failed three times as the job having left the queue, so a controller outage ended a healthy job's launch as JobStatusUnknown. They now treat it as still queued, as is_active does. * ci: run power_model and ubenchX tests and widen the path filter power_model and ubenchX tests ran in no workflow. The path filter listed a directory that does not exist and missed the workflow files, benchmark recipes and infx data files that tests read. Drop the two pytest paths that hold no tests. * fix(ci): post the recipe reminder on fork pull requests The pull_request token is read-only on forks, so the reminder failed on every fork PR. Run it on pull_request_target, which checks out no PR code, and page through all comments when looking for an earlier reminder. * fix(speedbench): fetch prepare.py at the reviewed commit The collectors piped NeMo-Skills prepare.py from its main branch into Python. Fetch the commit infx.bench.speedbench already pins, whose file is byte-identical today, so an upstream edit cannot change the prompt set or run new code in the collector. * fix(config): reject repeated keys in runners.yaml and referenced recipes Master configs already reject a key repeated within one mapping. The runner config and the recipes a config references were loaded with a plain safe loader, which silently keeps the last value. * fix(klaud): retry artifact downloads on transient failures Listings and reads already retry server errors, timeouts and transport failures. Artifact downloads did not, so one failed download read as a missing artifact and the planner dropped the holds that artifact records. * fix(bash): drop nounset from a recipe setup script and a collector default AGENTS.md forbids nounset and fallback defaults for caller-supplied configuration. The Muse Spark collector's draft model is not an input of speedbench-al.yml, so it becomes a constant. * test: delete assertions that read workflow and README text The test-quality rules require deleting tests that inspect checked-in text rather than running behavior. * refactor(recovery): remove the workflow-based recovery commands create-worktree, build-config and validate-workflow built a synthetic config for a dedicated recovery workflow. Recovery now goes through create-pr and the recovery guide forbids adding another workflow, so nothing calls them. * chore: delete files nothing references Ten 1k1k recipes and a chat template no config selects, a hugepage hook the cluster record does not run, a TensorRT-LLM patch script, an OperatorX consolidation script whose row key cannot hash current testlist args, and a waiver for a retired config whose patch scripts are gone. Also drop nounset from the pip setup script. * fix(ci): upload the same eval files on every retry The retry attempts listed SWE-bench outputs that the first attempt did not, in a different order in each template. Nothing produces those files any more, so drop them and keep one list per template. * refactor(workflows): share the primary sweep label list stage_results and merge_with_reuse each spelled out the three primary sweep labels. * docs: list video-generation in the README and state power_model's license README_zh.md already has the video-generation row. power_model declares GPL-3.0-only in its pyproject and source headers while the root README describes the repository as Apache-2.0. * fix(launch): let accounting end the wait when squeue stays down Waiting through a squeue outage with no bound could hold a runner after the job ended. When squeue keeps failing, a terminal state from accounting now ends the wait; a job neither source can place is still waited on. * fix(speedbench): keep the N/A guard to collectors that already announce it Six collectors run vllm bench serve without checking its status. Whether its exit code is non-zero on a usable pass is unverified, so leave them as they were. * docs(power_model): scope the license note to the root license * fix(merge): post /use instead of the legacy /reuse-sweep-run /reuse-sweep-run is deprecated but still recognized. The merge helper now posts the documented command. * revert: keep the 1k1k recipes and the DeepSeek-V4 chat template Recipes stay in the tree even when no config selects them.
Add pretraining/.gitkeep to track the root-level directory. Co-authored-by: Perplexity <noreply@perplexity.ai>
Add pretraining/torchtitan pinned to upstream commit 847fabbca08925078720beeafc174d2b1f8cbb66. Co-authored-by: Perplexity <noreply@perplexity.ai>
…rve with EP4/DCP8 KV-router points at concurrency 1-28 / [NVIDIA][AgentX] B200 Kimi-K3.1-Pro FP4 Dynamo+SGLang：以并发 1-28 的 EP4/DCP8 KV 路由配置点替换曲线 (SemiAnalysisAI#4964)
…仅在本仓库运行 (SemiAnalysisAI#5003) * ci: run scheduled jobs and PR bots only in this repository / 定时任务和 PR 机器人仅在本仓库运行 * fix: keep entries of the merged pull request and allow manual prompt warming / 保留所合并 PR 的条目并允许手动预热
…ng and the 1P1D PD recipe / [AMD][MI35X] Kimi K3.1 Pro sglang dflash agentic：统一镜像、P90 调优与 1P1D PD 配置 (SemiAnalysisAI#4928) * Image lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:5901db14 for every point * Single-node P90 tuning at concurrency 1 to 56, plus a concurrency 24 point * 1P1D PD at concurrency 128 and 144: TP8 DCP8 prefill and decode, KV over Mooncake RDMA, prefill HiCache * Single-node and PD publish as one sglang curve under one perf-changelog entry Co-authored-by: Kevin Mi <mikevin920@yahoo.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
… artifact 上传重试和 Python 项目设置抽为共享 action (SemiAnalysisAI#5011) * ci: share artifact upload retries and Python project setup as actions / 将 artifact 上传重试和 Python 项目设置抽为共享 action * test: temporary smoke workflow for the shared actions * test: remove the temporary smoke workflow
…fill + TileRT decode (TP8, 24 GPUs) (SemiAnalysisAI#4979) Adds dsv41flash-fp4-gb300-tilert-agentic-dspark-tp: DeepSeek-V4.1-Flash FP4 AgentX at concurrency 1 on GB300 NVL72. One vLLM TP4 prefill sends the prompt KV over NVLink to one TileRT decode, at TP8 and at 24 GPUs (three TP8 groups). Results are reused from run 42938502733. Original work by CrimsonDump in SemiAnalysisAI#4976. Co-authored-by: CrimsonDump <xbj0220ok@gmail.com> Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com>
…(agentx): 新增 GLM-5.2 MI355X ATOM PD 分离式测试点 (SemiAnalysisAI#4045)
…at TP8 c6 (SemiAnalysisAI#4983) * Rerun Muse Spark MI355X SGLang AgentX with stream interval 50 at TP8 c6 Lower the TP8 concurrency 6 stream interval from 128 to 50 and append a changelog entry that reruns every point of musespark-fp4-mi355x-sglang-agentic-dflash on the same image. * Set the changelog PR link
… / 用合并提交的基线父提交校验 CollectiveX changelog (SemiAnalysisAI#5020)
…00 Kimi-K3.1-Pro FP4 vLLM 聚合式配置 (SemiAnalysisAI#4871) * Update Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs * Link the B300 vLLM changelog entry to its PR * Update Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs * Update Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs * Update Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs * Update Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs Add concurrency 6 and 16 to the TP8 x DCP8 recipe. Run concurrency 24, 40 and 48 on the SemiAnalysisAI#4878 recipe, which also graphs prefill and mixed steps up to 4096 tokens, without the TEP shared-expert variable. Keep concurrency 64 on EP8 MegaMoE with max-num-batched-tokens 16384. Co-authored-by: Bowen Wang <bowen@inferact.ai> * Tune Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs Admit prefill once every 8 engine steps (prefill-schedule-interval 8) at concurrency 24, 40, 48, 64 and 72, so more steps run the fused decode. Move concurrency 48 to the EP8 MegaMoE recipe with FULL_DECODE_ONLY graphs, raise gpu-memory-utilization to 0.98 at concurrency 64, and add concurrency 72. Signed-off-by: Mi Sun Park <misunp@nvidia.com> * Add concurrency 20 to the Kimi-K3.1-Pro B300 vLLM AgentX configs Run concurrency 20 on the mixed-graph recipe with the concurrency-24 settings: TP8 x DCP8, DSpark drafting 3 tokens, CUDA graphs for prefill and mixed steps up to 4096 tokens, and prefill-schedule-interval 8. Signed-off-by: Mi Sun Park <misunp@nvidia.com> * Narrow the Kimi-K3.1-Pro B300 vLLM AgentX sweep to seven frontier points Sweep concurrency 1, 6, 12, 24, 48, 64 and 72. The dropped points (4, 8, 16, 20 and 40) sit on or below the line between their neighbours, so the curve keeps its shape with fewer runs. Concurrency 12 moves to the mixed-graph recipe with the concurrency-24 settings: TP8 x DCP8, DSpark drafting 3 tokens, CUDA graphs for prefill and mixed steps up to 4096 tokens, and prefill-schedule-interval 8. On the same image this raised P90 interactivity by about 10 % at the same throughput per GPU as the FULL_DECODE_ONLY recipe at concurrency 12. Signed-off-by: Mi Sun Park <misunp@nvidia.com> * Lower Kimi-K3.1-Pro B300 concurrency-72 gpu-memory-utilization to 0.97 The concurrency-72 eval job's acceptance-length gate sends 80 requests at once. At gpu-memory-utilization 0.98 the resulting prefill step overflowed the direct DCP workspace, and the fallback combine ran out of GPU memory on every rank, so the eval could not run. Use 0.97, as at concurrency 48. Signed-off-by: Mi Sun Park <misunp@nvidia.com> * Drop Kimi-K3.1-Pro B300 concurrency 72; run concurrency 64 at 0.97 Concurrency 72 had a P90 TTFT of 10.7 s, about four times the other points, for 2 % more throughput per GPU than concurrency 64. Its TTFT tail is engine queue time while requests wait for GPU KV cache, not prefill or preemption, so no setting within this recipe brings it under a few seconds. Concurrency 64 becomes the highest point. The agentic eval now runs at concurrency 64, so it takes gpu-memory-utilization 0.97 like concurrency 48: at 0.98 the eval's 80-request acceptance-length burst ran out of memory in the DCP combine fallback. At 0.97, concurrency 64 measured the same throughput per GPU as at 0.98. Signed-off-by: Mi Sun Park <misunp@nvidia.com> * perf-changelog: describe the Kimi-K3.1-Pro B300 sweep as six points [skip-sweep] Concurrency 1 is not on the throughput/E2EL frontier, so drop 'frontier' from the entry. Signed-off-by: Ankur Singh <ankusingh@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mi Sun Park <misunp@nvidia.com> Signed-off-by: Ankur Singh <ankusingh@nvidia.com> Co-authored-by: Bowen Wang <bowen@inferact.ai> Co-authored-by: Ankur Singh <ankusingh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Move the shared srt-slurm submodule from v2.66.2 (acfb4dc) to 2d9079c (v2.69.1 plus NVIDIA/srt-slurm#666) and update the pinned documentation links. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…MiniMax-M3.1-Flash Dynamo vLLM curves (SemiAnalysisAI#4998)
…r more UCCL variables / CollectiveX：打印日志尾部时屏蔽主机与地址，并多清除四个 UCCL 变量 (SemiAnalysisAI#5041) * CollectiveX: mask hosts in printed log tails and scrub four more UCCL variables A failed step prints the last 100 lines of its log, and in CI that lands in a public job log. The tail now passes through runtime/redact.py, which masks IPv4, IPv6 and MAC addresses and the submit host and allocation node names (also as Slurm ranges and with a domain or interface suffix). If masking fails the tail is withheld. Ranks inherit the launcher's environment, and UCCL reads UCCL_RDMA_ADAPTIVE_SLEEP, UCCL_EP_TRANSPORT, UCCL_EP_DISPATCH_CONFIG and UCCL_EP_COMBINE_CONFIG at run time, so a value left in the launching shell changed the run under an unchanged label. They are now unset with the other UCCL variables. * CollectiveX: mask kernel host names that differ from Slurm node names
…ectiveX：pair period 计时不再在 pair 之间插入 event (SemiAnalysisAI#5039) * fix(collectivex): time the pair period without events between pairs The published pair period was a window between two CUDA events around one dispatch+combine pair. An event node costs device time of its own, so the window read above what back-to-back pairs cost: 4.5-13.5 us at T=1 on NVIDIA EP8 and about 6 us on AMD, roughly constant in T. Each chain trial now runs a third sibling, the span chain, with one event before its first kept pair and one after its last. Its time per pair sets the period's level; the pair windows, shifted onto it per trial, keep the pair-to-pair spread. chain_health.window_overhead_us reports what the windows read above the span, and the component's origin becomes chained-span-median. Also: - swap-blocks profiles take 40 iterations per point instead of 100; - an error that is not finite (NaN output, shape mismatch) is written as null beside its failed verdict instead of crashing the result writer; - methodology: the chain floors can sum to more than the pair period, and bandwidth is a per-GPU rate. * chore(collectivex): add the changelog entries for the pair period and swap-blocks changes * fix(collectivex): release the span graph before the other chain graphs are captured Keeps at most two chain graphs alive at once, as before the span chain was added.
…cation / CollectiveX：中断 run.sh 时结束运行并释放分配 (SemiAnalysisAI#5042) * CollectiveX: an interrupted run.sh ends the run and releases its allocation Ctrl-C on run.sh only skipped to the next shard, and a signal during a queued salloc or a running case left the Slurm job behind. run.sh now waits on each shard in the background, so its HUP/INT/TERM trap runs at once: it stops the launcher and the command the launcher is running, waits for the launcher to cancel its own job, and starts no further shard. The launcher waits on the case step in the background for the same reason, stops salloc and the step before cleanup, finds a job that was still queued through the scheduler log, and ignores repeat signals while it cleans up. * CollectiveX: an interrupt reaches every process under the shard TERM went to the launcher and its direct children only, so a login-node image import two levels down kept running after run.sh exited. Comments trimmed.
* feat(minimaxm31flash-mi355x-vllm): minimax-m31-flash-75bc8cf1 image and its serving recipe Pin vllm/vllm-openai-rocm:minimax-m31-flash-75bc8cf1 by digest. The image turns on its amd_kernels routes (GEMM, MoE, qkv, attention, indexer, norm, draft, sampling, top-p, SP prefill) by default, so the recipe drops the nightly's env switches and serves every point on the NVFP4 KV cache with DBO, breakable CUDA graphs up to 2048 tokens and 8 renderer workers. - c1-c16: DSpark 7, GPU KV only; the c1-c8 mono decode lane is removed. - c32-c64: SimpleCPUOffload tier (1349 GB) with LRU eviction; c32 moves to the dram search-space row, c64 keeps DSpark 3. - Batch-sharded sampling from c2 (vLLM refuses it below max-num-seqs 4 at TP4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: benzhuzhu <zhutaoyu97@gmail.com> * fix(srt-slurm): drop patches 901 and 902, which the 2d9079c pin already includes The srt-slurm pin 2d9079c contains NVIDIA/srt-slurm#664 (TileRT multi-node decode) and SemiAnalysisAI#666 (post_eval.container_image), so both patches fail to apply and every srt launch stops in the checkout step. Same deletion as SemiAnalysisAI#5019. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: benzhuzhu <zhutaoyu97@gmail.com> --------- Signed-off-by: benzhuzhu <zhutaoyu97@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
rltakashige
requested review from
Oseltamivir,
functionstackx and
hbarclay
as code owners
November 12, 2026 19:31
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merge InferenceX
main(111 commits, up to8a6c3ccdc) into this repository, asdocs/LOCAL.mddescribes.Merge this with a merge commit. A squash or rebase drops the merge parent, and the next merge then conflicts with every change in this one.
Details
infx/srt_slurm/generate.pykeeps the skip of serve-path keys, with upstream's newgenerate_config_matrixcall.infx/srt_slurm/single_node.pytakes upstream's rewrittenselect_recipe, with thenative_powerargument added back.perf-changelog.yamlandCODEOWNERSkeep this repository's copies, so the merge starts no sweep.2d9079c6. The three patch files apply to it unchanged.AI model disclosure
claude-fable-5-2: used for resolving the conflicts and testing.PR checklist and change type
Type of Change
mainChecklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概述
按
docs/LOCAL.md的说明，将 InferenceX 的main（111 个 commit，截至8a6c3ccdc）合并到本仓库。请使用 merge commit 合并。squash 或 rebase 会丢掉合并的父提交，下一次合并就会与本次的所有改动冲突。
详情
infx/srt_slurm/generate.py保留了对 serve 路径配置键的跳过，并采用上游新的generate_config_matrix调用。infx/srt_slurm/single_node.py采用上游重写后的select_recipe，并重新加入native_power参数。perf-changelog.yaml和CODEOWNERS保留本仓库的版本，因此这次合并不会触发 sweep。2d9079c6。三个 patch 文件无需修改即可应用。AI 模型使用说明
claude-fable-5-2：用于解决冲突和测试。改动类型
其他：合并上游
main。