Skip to content

[Klaud Cold] kimik2.5-fp4-b300-vllm: v0.26.0 image, extend conc to 512, add DEP4 arm / 升级 B300 镜像至 v0.26.0，扩展并发，新增 DEP4 #2442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xinli-sw wants to merge 12 commits into
base: main
Choose a base branch
from
Open

[Klaud Cold] kimik2.5-fp4-b300-vllm: v0.26.0 image, extend conc to 512, add DEP4 arm / 升级 B300 镜像至 v0.26.0，扩展并发，新增 DEP4 #2442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
22f4d7f
kimik2.5 fp4 b300: v0.26.0 image, TP/DEP sweep, tuning
xinli-sw Aug 2, 2026
dfa9591
kimik2.5 fp4 b300: v0.26.0 image, TP/DEP sweep, tuning
xinli-sw Aug 2, 2026
7746671
kimik2.5 fp4 b300: nightly image, extend sweep space with TP/DEP arms
xinli-sw Aug 2, 2026
129b9b1
drop --attention-config from B300
xinli-sw Aug 3, 2026
80d9ed8
add attention-config with prefill query quant and linear-backend flas…
xinli-sw Aug 3, 2026
20a5a28
set max-cudagraph-capture-size to 2x concurrency
xinli-sw Aug 3, 2026
67892db
set max-num-batched-tokens to 16k
xinli-sw Aug 3, 2026
b13329c
remove max-num-batched-tokens, set VLLM_USE_V2_MODEL_RUNNER=0
xinli-sw Aug 3, 2026
aa76f27
add --no-enable-flashinfer-autotune
xinli-sw Aug 4, 2026
51c8c80
merge origin/main, re-append changelog
xinli-sw Aug 4, 2026
c135460
fix perf-changelog
xinli-sw Aug 4, 2026
53081ea
remove --no-enable-flashinfer-autotune, add TEP arms
xinli-sw Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b300.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,21 @@ check_env_vars \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
GMU=0.90
PREFILL_SCHEDULE_ARGS=()
if [ "${DP_ATTENTION:-false}" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
GMU=0.85
PREFILL_SCHEDULE_ARGS=(--prefill-schedule-interval 4)
fi

EP_ARGS=()
if [ "${EP_SIZE:-1}" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
fi
Comment on lines +19 to +31

@claude claude Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The top-of-file comment in kimik2.5_fp4_b300.sh (lines 3-5) and the matching comment in configs/nvidia-master.yaml (lines 1526-1528) both say the B300 recipe 'reuses the existing Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available' — but this PR adds exactly that B300-specific tuning (DP_ATTENTION/EP handling, prefill-schedule-interval, fp8 kv-cache, attention-config, linear-backend, new env vars), so the comment is now stale and misleading. Purely a documentation issue — recommend removing or updating both comments.

Extended reasoning...

What the bug is: The unmodified top-of-file comment in benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b300.sh (lines 3-5) states:

does not have a B300-specific recipe, so this script reuses the existing Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available.

The identical claim is duplicated in configs/nvidia-master.yaml at lines 1526-1528, directly above the kimik2.5-fp4-b300-vllm config block. Neither comment was touched by this PR, but this PR is precisely what makes them false.

Why it's now false: Comparing the new kimik2.5_fp4_b300.sh vllm serve invocation against kimik2.5_fp4_b200.sh, the two scripts diverge substantially after this PR:

  • B300 adds a DP_ATTENTION branch that switches --tensor-parallel-size/--data-parallel-size, adjusts GPU memory utilization (0.90 -> 0.85), and adds --prefill-schedule-interval 4; B200 has none of this.
  • B300 adds EP_ARGS -> --enable-expert-parallel; absent in B200.
  • B300 adds --kv-cache-dtype fp8, --attention-config '{\"mla_prefill_backend\":\"FLASHINFER\",...}', and --linear-backend flashinfer_cutlass; none of these exist in B200.
  • B300's --max-cudagraph-capture-size is now $((CONC*2)) vs B200's fixed 2048, and --stream-interval is 32 vs B200's 20.
  • B300 sets new env vars VLLM_USE_V2_MODEL_RUNNER=0, VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS, VLLM_RPC_TIMEOUT; B200 instead sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, which B300 does not.

So the B300 recipe is no longer a copy-as-is of B200 — it is a distinct, B300-tuned recipe, and that tuning is exactly what the comment says doesn't exist yet.

Code path / proof: A reader opening kimik2.5_fp4_b300.sh sees the comment at lines 3-5 claiming 'reuses B200 as-is,' then scrolls a few lines down (post-PR, lines 19-31 and 67-86) and sees DP_ATTENTION branching, EP handling, fp8 kv-cache, attention-config, and linear-backend flags that have no B200 counterpart. The comment directly contradicts the code immediately below it. The same applies to the nvidia-master.yaml comment at lines 1526-1528, sitting right above the kimik2.5-fp4-b300-vllm: block that this PR also modifies (new nightly image, new DEP4 search-space arm).

Why existing code/review doesn't catch this: This is a plain-text comment, not something any linter, test, or CI check inspects — it can only be caught by manual review comparing the comment's claim against the diff.

Impact: Purely a documentation/maintainability issue — nothing breaks functionally. But a future engineer syncing B300 with B200, or auditing why the recipes differ, would be misled into thinking B300 has no tuning beyond a straight copy, when in fact it now has meaningfully different parallelism, quantization, and scheduling logic.

Fix: Remove or rewrite both comments (script header and yaml block comment) to reflect that B300-specific tuning has been added, rather than stating it 'reuses B200 as-is.'



# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
Expand All @@ -37,6 +52,9 @@ nvidia-smi

export TORCH_CUDA_ARCH_LIST="10.0"
export PYTHONNOUSERSITE=1
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=""
export VLLM_RPC_TIMEOUT=600000

SERVER_LOG=/workspace/server.log

Expand All @@ -49,13 +67,20 @@ start_gpu_monitor

set -x
vllm serve $MODEL_PATH --served-model-name $MODEL --host 0.0.0.0 --port $PORT \
--tensor-parallel-size $TP \
--gpu-memory-utilization 0.90 \
"${PARALLEL_ARGS[@]}" \
"${EP_ARGS[@]}" \
"${PREFILL_SCHEDULE_ARGS[@]}" \
--gpu-memory-utilization "$GMU" \
--max-model-len $MAX_MODEL_LEN \
--max-num-seqs $CONC \
--reasoning-parser kimi_k2 \
--tool-call-parser kimi_k2 \
--compilation_config.pass_config.fuse_allreduce_rms true \
--kv-cache-dtype fp8 \
--max-cudagraph-capture-size "$((CONC * 2))" \
--stream-interval 32 \
--attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
--linear-backend flashinfer_cutlass \
--no-enable-prefix-caching \
--trust-remote-code > $SERVER_LOG 2>&1 &

Expand Down
25 changes: 22 additions & 3 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1436,7 +1436,7 @@ kimik2.5-fp4-b200-vllm:
# Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available.

kimik2.5-fp4-b300-vllm:
image: vllm/vllm-openai:v0.22.0
image: vllm/vllm-openai:nightly-e2fa28594f7baad142a426b0b6a2cfe2c79201c7
model: nvidia/Kimi-K2.5-NVFP4
model-prefix: kimik2.5
runner: b300
Expand All @@ -1448,8 +1448,27 @@ kimik2.5-fp4-b300-vllm:
- isl: 8192
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 1, conc-end: 4 }
- { tp: 4, ep: 1, conc-start: 1, conc-end: 128 }
- { tp: 8, ep: 1, conc-list: [1] }
- { tp: 4, ep: 1, conc-start: 1, conc-end: 512 }
- { tp: 8, ep: 8, dp-attn: false, conc-list: [1] }
- { tp: 4, ep: 4, dp-attn: false, conc-start: 1, conc-end: 512 }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512 }
-vllm:
image: vllm/vllm-openai:nightly-e2fa28594f7baad142a426b0b6a2cfe2c79201c7
model: nvidia/Kimi-K2.5-NVFP4
model-prefix: kimik2.5
runner: b300
precision: fp4
framework: vllm
multinode: false
scenarios:
fixed-seq-len:
- isl: 8192
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-list: [1] }
- { tp: 4, ep: 1, conc-start: 1, conc-end: 512 }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512 }

dsr1-fp8-b200-sglang-mtp:
image: lmsysorg/sglang:v0.5.12-cu130
Expand Down
6 changes: 6 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5449,3 +5449,9 @@
- "Enable prefill-only INT4 quick-reduce: set VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 and VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=2048 on the prefill workers via a new prefill_env channel (mirrors the existing decode_env path in server_vllm.sh)."
- "Cap the 1P1D TP4 concurrency sweep at 256 (was 512); drop the 2P1D TP4 layout (128/256/512) as it is CI-flaky with negligible curve impact."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1943

- config-keys:
- kimik2.5-fp4-b300-vllm
description:
- "Kimi K2.5 NVFP4 B300 vLLM: nightly image, extend sweep space with TP/DEP arms, TP8 conc-1 only, DEP gmu 0.85"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2442
4 changes: 3 additions & 1 deletion runners/launch_b300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,9 @@ else
# MODEL stays as the HF id for the client (--served-model-name, tokenizer);
# MODEL_PATH is what the server reads weights from.
MODEL_BASENAME="${MODEL##*/}"
if [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}"
Comment on lines 408 to 412

@claude claude Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new staged-model bypass in launch_b300-nv.sh keys only on MODEL_PREFIX=="kimik2.5" && PRECISION=="fp4", so it also matches the unrelated kimik2.5-fp4-b300-vllm-agentic and kimik2.5-fp4-b300-vllm-agentic-mtp configs, forcing their already-staged models (Kimi-K2.5-NVFP4, Kimi-K2.6-NVFP4) off the read-only pre-staged mount and onto WRITABLE_MODELS_DIR. For the mtp config, MODEL is the bare string Kimi-K2.6-NVFP4 (no HF org prefix), so if that path isn't already populated the forced hf download will fail outright since it isn't a valid HF repo id. The condition should key off the specific model basename or config name (e.g. kimik2.5-fp4-b300-vllm) instead of the broad prefix+precision pair.

Extended reasoning...

The new branch added at runners/launch_b300-nv.sh (single-node else-branch, ~line 409) is:

if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
    export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
    export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}"
...

It's inserted before the STAGED_MODELS membership check, and it keys purely on MODEL_PREFIX+PRECISION, not on the actual model being benchmarked or the specific config. The PR's intent (per the diff/description) is to force the one target config, kimik2.5-fp4-b300-vllm (model nvidia/Kimi-K2.5-NVFP4), onto the writable dir — presumably to pick up a fresh nightly image/weights rather than a stale pre-staged snapshot. But two other, unrelated configs in configs/nvidia-master.yaml share the same model-prefix: kimik2.5 / precision: fp4 pair and are unintentionally swept into the new branch:

  • kimik2.5-fp4-b300-vllm-agentic (line 1580): model: nvidia/Kimi-K2.5-NVFP4, runner: cluster:b300-nv, multinode: false.
  • kimik2.5-fp4-b300-vllm-agentic-mtp (line 1600): model: Kimi-K2.6-NVFP4, runner: cluster:b300-nv, multinode: false.

Both route through the same single-node else-branch of launch_b300-nv.sh (the launcher is resolved generically from the runner name, and launch_b300-nv.sh is the only b300 launcher script), so both hit the new condition.

Why this matters — concrete proof:

  1. STAGED_MODELS (defined a few lines above, line ~392 in the file) includes both Kimi-K2.5-NVFP4 and Kimi-K2.6-NVFP4. Before this PR, both agentic configs' MODEL_BASENAME matched that list and resolved to the read-only pre-staged mount: MODEL_PATH=/scratch/models/Kimi-K2.5-NVFP4 and /scratch/models/Kimi-K2.6-NVFP4 respectively.
  2. After this PR, the new condition fires first for both (since MODEL_PREFIX=kimik2.5 and PRECISION=fp4 for both), so MODEL_PATH becomes /data/models/Kimi-K2.5-NVFP4 and /data/models/Kimi-K2.6-NVFP4 — the writable dir — regardless of whether anything is staged there.
  3. The downstream bench script (same idempotent-download pattern as kimik2.5_fp4_b300.sh: if [[ ! -d \"$MODEL_PATH\" || -z \"$(ls -A \"$MODEL_PATH\")\" ]]; then hf download \"$MODEL\" --local-dir \"$MODEL_PATH\"; fi) will trigger a fresh hf download if /data/models/... isn't already populated for these models.
  4. For kimik2.5-fp4-b300-vllm-agentic, MODEL=nvidia/Kimi-K2.5-NVFP4 is at least a valid HF repo id, so this 'only' costs an unnecessary multi-hundred-GB re-download of a model that was already sitting on the read-only staged mount.
  5. For kimik2.5-fp4-b300-vllm-agentic-mtp, MODEL=Kimi-K2.6-NVFP4 has no HF org/namespace prefix — it is not a resolvable HF repo id at all. If /data/models/Kimi-K2.6-NVFP4 doesn't happen to already exist from some unrelated prior run, hf download Kimi-K2.6-NVFP4 --local-dir ... will fail outright, breaking that recipe's sweep run entirely.

Existing code doesn't prevent this because the new condition was added as a plain prefix/precision predicate rather than being scoped to the specific model basename (MODEL_BASENAME == \"Kimi-K2.5-NVFP4\") or the specific config/recipe name, so it can't distinguish the PR's intended target from any other kimik2.5/fp4 recipe that happens to share those two fields.

Fix: scope the new branch to the actual model basename (e.g. [[ $MODEL_BASENAME == \"Kimi-K2.5-NVFP4\" ]]) or to the specific config name being run, so it only affects kimik2.5-fp4-b300-vllm and leaves the unrelated agentic/agentic-mtp recipes on their existing staged-model resolution path.

else
export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
Expand Down
Loading