From f2f5a0e892b0352e301517f204896500c5e09eb6 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 1 Aug 2026 22:44:01 -0400 Subject: [PATCH 1/2] glm5.2-fp4-b300-sglang-agentic-mtp: GLM-5.2 B300 AgentX with EAGLE MTP and golden-AL simulated acceptance Adds the spec-decode arm of the GLM-5.2 NVFP4 B300 agentic-coding recipe, the arm that replaces the STP entry once agentic non-MTP is deprecated for GLM-5.2 on 2026-08-03 (MODELS.md). - benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh: the validated STP script plus SGLang EAGLE off GLM-5.2's built-in nextn head (num-steps 3, eagle-topk 1, 4 draft tokens) and SGLANG_SIMULATE_ACC_* pinned to the golden AL, left off under EVAL_ONLY so accuracy runs keep real verification. - golden_al_distribution/glm5.2_mtp.yaml: the GLM-5.2 MTP curve verbatim from the speedbench-al.yml artifact of run 28058352479, plus the README rows. - configs/nvidia-master.yaml: TP8 + HiCache at conc [1, 4, 8, 12, 16] on lmsysorg/sglang:v0.5.16-cu130, the first release that reads SGLANG_SIMULATE_ACC_TOKEN_MODE. Co-Authored-By: Claude Opus 5 (1M context) --- .../agentic/glm5.2_fp4_b300_sglang_mtp.sh | 288 ++++++++++++++++++ configs/nvidia-master.yaml | 33 ++ golden_al_distribution/README.md | 1 + golden_al_distribution/README_zh.md | 1 + golden_al_distribution/glm5.2_mtp.yaml | 19 ++ perf-changelog.yaml | 12 + 6 files changed, 354 insertions(+) create mode 100755 benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh create mode 100644 golden_al_distribution/glm5.2_mtp.yaml diff --git a/benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh b/benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh new file mode 100755 index 0000000000..2b669741e1 --- /dev/null +++ b/benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh @@ -0,0 +1,288 @@ +#!/usr/bin/env bash +set -euo pipefail +set -x + +# Agentic trace replay benchmark for GLM-5.2 NVFP4 on B300 using SGLang with +# EAGLE/MTP speculative decoding. +# +# Spec-decode-only sibling of agentic/glm5.2_fp4_b300_sglang.sh, per the AgentX +# policy that agentic recipes are run and published with speculative decoding +# enabled rather than as an STP/MTP A/B (MODELS.md: GLM-5.2 agentic non-MTP is +# deprecated after 2026-08-03). Everything outside the SPEC_ARGS and +# SGLANG_SIMULATE_ACC_* blocks is the validated STP script unchanged. +# +# Server flags follow the SGLang cookbook B300 NVFP4 single-node recipes +# (https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2): +# DP_ATTENTION=false -> low-latency arm (TP8, fp8 KV, cutedsl bf16 GEMM) +# DP_ATTENTION=true -> high-throughput DEP arm (TP8 + DP8 attention-DP + +# EP_SIZE expert-parallel MoE via --ep-size) +# Only the low-latency arm is wired into the master config for this MTP recipe +# (see the entry comment on glm5.2-fp4-b300-sglang-agentic-mtp); the DEP branch +# is kept intact so the throughput arm can be added without re-deriving it. +# +# Required env vars: +# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, +# EP_SIZE, DP_ATTENTION +# +# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache. + +source "$(dirname "$0")/../../benchmark_lib.sh" + +check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION + +if [[ -n "${SLURM_JOB_ID:-}" ]]; then + echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" +fi + +# `hf download` creates the target dir if missing and is itself idempotent. +# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE. +# Either way, MODEL_PATH is what the server is launched with. +if [[ -n "${MODEL_PATH:-}" ]]; then + if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then + hf download "$MODEL" --local-dir "$MODEL_PATH" + fi +else + hf download "$MODEL" + export MODEL_PATH="$MODEL" +fi +nvidia-smi + +resolve_trace_source +install_agentic_deps + +SERVER_LOG="$RESULT_DIR/server.log" +mkdir -p "$RESULT_DIR" + +CACHE_ARGS=() +if require_agentic_kv_offload_backend hicache; then + # HiCache extends RadixAttention: prefixes evicted from the HBM KV pool + # spill to a pinned host pool instead of being recomputed. On the + # 1M-context agentic corpus the live working set outgrows HBM past + # conc 8 (TP8) / 64 (DP8) and the radix hit rate collapses to <0.1 + # against a ~0.97 theoretical ceiling, so every turn re-prefills its + # whole history; the host tier restores those hits at C2C bandwidth. + # GLM-5.2 is DSA/MLA-family (attention_backend=dsa): every rank holds + # complete per-token KV (169.98 GB device pool per rank, replicated on + # all 8 ranks), so host capacity is controlled through the host/device + # token-capacity ratio like the DSv4 recipe, NOT a per-rank + # --hicache-size. A GB-based size of TOTAL_CPU_DRAM_GB/TP pinned the + # whole 0.80-DRAM budget (8 x 299 GB) at init on top of 465 GB of + # weights and OOM-killed the node (run 29678598595); DSv4's own + # ratio=2 default pins 2 x 170 GB x 8 = 2.7 TB here and OOMs too + # (GLM-5.2's device pool is far larger than DSv4's). Fractional 0.75 + # = ~128 GB/rank = ~1.0 TB total, matching the cluster's proven ~1 TB + # host-pool envelope; validated on-node 2026-07-19 (boot + 4.2M-token + # overflow bench forcing eviction through the DSA KV+INDEXER pools). + # The ratio is relative to the device pool, so MTP's slightly smaller + # device pool (the nextn layer takes its own KV) only shrinks it. + DEFAULT_HICACHE_RATIO=0.75 + HICACHE_RATIO="${HICACHE_RATIO:-$DEFAULT_HICACHE_RATIO}" + if awk -v r="$HICACHE_RATIO" -v cap="$DEFAULT_HICACHE_RATIO" 'BEGIN { exit !(r > cap) }'; then + echo "Error: HICACHE_RATIO=$HICACHE_RATIO exceeds configured limit $DEFAULT_HICACHE_RATIO" >&2 + exit 1 + fi + HICACHE_WRITE_POLICY="${HICACHE_WRITE_POLICY:-write_back}" + HICACHE_IO_BACKEND="${HICACHE_IO_BACKEND:-direct}" + HICACHE_MEM_LAYOUT="${HICACHE_MEM_LAYOUT:-page_first_direct}" + echo "HiCache CPU tier: ratio=$HICACHE_RATIO, capacity=${TOTAL_CPU_DRAM_GB} GB, write_policy=$HICACHE_WRITE_POLICY, io_backend=$HICACHE_IO_BACKEND, mem_layout=$HICACHE_MEM_LAYOUT" + CACHE_ARGS=( + --enable-hierarchical-cache + --hicache-ratio "$HICACHE_RATIO" + --hicache-write-policy "$HICACHE_WRITE_POLICY" + --hicache-io-backend "$HICACHE_IO_BACKEND" + --hicache-mem-layout "$HICACHE_MEM_LAYOUT" + ) +fi + +# With attention-DP, front the DP ranks with sglang-router using consistent +# hashing on the AIPerf correlation id so multi-turn sessions stay on the DP +# rank that holds their radix-cache prefix. +USE_SGLANG_ROUTER=false +SGLANG_BACKEND_PORT="$PORT" +ROUTER_LOG="$RESULT_DIR/router.log" +if [ "$DP_ATTENTION" = "true" ]; then + USE_SGLANG_ROUTER=true + export AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true + SGLANG_BACKEND_PORT=$((PORT + 1)) + SGLANG_ROUTER_METRICS_PORT=$((PORT + 10000)) +fi + +# MTP: GLM-5.2 ships its own nextn head, so EAGLE runs off the checkpoint with +# no external draft model. num-steps 3 / eagle-topk 1 / num-draft-tokens 4 is +# 3 speculative tokens per verification step -- the same shape the GLM-5.2 +# GB300 dynamo-sglang agentic recipes use, and the draft length whose golden AL +# is pinned below. +SPEC_ARGS=( + --speculative-algorithm EAGLE + --speculative-num-steps 3 + --speculative-eagle-topk 1 + --speculative-num-draft-tokens 4 +) + +PARALLEL_ARGS=(--tp "$TP" --ep-size "$EP_SIZE") +CHUNKED_PREFILL_SIZE=8192 +if [ "$DP_ATTENTION" = "true" ]; then + # chunked-prefill-size is a whole-engine budget split across DP ranks: + # the cookbook HT cell's 8192 becomes 1,024 tokens/rank/step under dp8, + # which starves prefill on the 1M-context agentic corpus (observed: a + # conc-256 warmup could not drain within AIPerf's 1800s grace period + # while KV usage sat at ~0.01). Use the cookbook's own dp8 lever from + # the B200 cells (32768 = ~4096/rank). + CHUNKED_PREFILL_SIZE=32768 + PARALLEL_ARGS+=( + --dp "$TP" + --enable-dp-attention + --tokenizer-worker-num "$TP" + --dist-init-addr "127.0.0.1:$((PORT + 2000))" + ) + # GLM-5.2-NVFP4 leaves the MTP/nextn layer unquantized (hf_quant_config + # excludes model.layers.78*), so the EAGLE draft MoE is bf16 and + # UnquantizedFusedMoEMethod pins it to the triton runner core. Inheriting + # the target model's FlashInfer all-to-all then has no (flashinfer, triton) + # pre-permute and the engine dies at init with "Pre-permute function for + # flashinfer to triton is not registered". SGLang handles this in + # _deepseek_spec_moe_resolution but gates the hook on is_hip(), so on CUDA + # the draft silently inherits; set upstream's own ROCm values explicitly. + # Only needed once expert parallelism puts an a2a in the MoE path -- the + # plain-TP arm below has none. + SPEC_ARGS+=( + --speculative-moe-a2a-backend none + --speculative-moe-runner-backend triton + ) +else + # Cookbook low-latency levers; the DP-attention cell omits them. + PARALLEL_ARGS+=( + --kv-cache-dtype fp8_e4m3 + --bf16-gemm-backend cutedsl + --max-prefill-tokens 8192 + ) +fi + +# AgentX concurrency counts live session trees, not individual requests. +# Allow subagent fan-out to exceed CONC without clipping request bursts. +MAX_RUNNING_REQUESTS=$((2 * CONC)) +GRAPH_ARGS=() +if [ "$DP_ATTENTION" != "true" ]; then + # Cookbook low-latency captures graphs up to its request cap; the + # DP-attention cell leaves the CUDA-graph batch list at SGLang defaults. + # --cuda-graph-max-bs counts requests, not verification tokens: SGLang's + # spec-decode graph runner scales each captured batch by + # --speculative-num-draft-tokens itself. + CUDA_GRAPH_MAX_BS=$MAX_RUNNING_REQUESTS + [ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64 + GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS") +fi + +export PYTHONNOUSERSITE=1 +export TORCH_CUDA_ARCH_LIST=10.0 +# Agentic warmup dispatches hundreds of large prompts at once; allow up to +# 15 minutes of TCP progress before AIPerf declares a connection dead. +export AIPERF_HTTP_TCP_USER_TIMEOUT=900000 +# AIPerf pins one pooled keep-alive connection per session (client-side +# keep-alive 300s) while uvicorn's default SGLANG_TIMEOUT_KEEP_ALIVE is 5s; +# inter-turn idle gaps (capped at 10s) can reuse a socket exactly as the +# server closes it -> ECONNRESET -> terminal warmup failure. Outlast the +# client pool so the race cannot occur. +export SGLANG_TIMEOUT_KEEP_ALIVE=900 + +# AgentX pins acceptance to the committed golden AL so submissions are compared +# on system performance at a fixed acceptance target rather than on draft-head +# quality (golden_al_distribution/README.md). 2.99 is the GLM-5.2 curve at +# num_speculative_tokens=3, thinking_on +# (golden_al_distribution/glm5.2_mtp.yaml, SPEED-Bench coding, run 28058352479). +# One curve per model: it was collected on the FP8 checkpoint, and the NVFP4 +# checkpoint ships the same nextn head. +# +# SGLANG_SIMULATE_ACC_TOKEN_MODE only exists from SGLang v0.5.16, which is why +# this recipe pins v0.5.16-cu130 rather than the STP sibling's v0.5.15.post1 -- +# an older image would silently honor ACC_LEN/ACC_METHOD and ignore the +# token-mode half of the contract. +# +# EVAL_ONLY leaves simulated acceptance off: it commits drafted tokens +# regardless of the target logits, so generated text is wrong and the eval +# would score ~0. +if [ "${EVAL_ONLY:-false}" != "true" ]; then + export SGLANG_SIMULATE_ACC_LEN=2.99 + export SGLANG_SIMULATE_ACC_METHOD=match-expected + export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token +fi + +SGLANG_CMD=( + python3 -m sglang.launch_server + --model-path "$MODEL_PATH" + --served-model-name "$MODEL" + --host 0.0.0.0 + --port "$SGLANG_BACKEND_PORT" + --trust-remote-code + "${PARALLEL_ARGS[@]}" + --quantization modelopt_fp4 + # GLM-5.2 emits the GLM-4.7-style // format; + # the glm47 parser is required for structured message.tool_calls (glm45 + # leaves calls as raw text). Without it the SWE-bench mini-swe-agent eval + # dies with RepeatedFormatError ("No tool calls found in the response") on + # every instance and scores 0. Reasoning parser keeps hybrid-thinking + # output in reasoning_content instead of polluting content. Neither flag + # affects trace-replay throughput (pre-canned replay discards live + # responses). + --tool-call-parser glm47 + --reasoning-parser glm45 + --chunked-prefill-size "$CHUNKED_PREFILL_SIZE" + --mem-fraction-static 0.85 + --max-running-requests "$MAX_RUNNING_REQUESTS" + "${SPEC_ARGS[@]}" + "${GRAPH_ARGS[@]}" + "${CACHE_ARGS[@]}" + --watchdog-timeout 1800 + --enable-metrics +) + +printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt" +printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt" + +{ + echo "=== SGLANG_SIMULATE_ACC_* env vars at launch (empty => real verification) ===" + env | grep -E '^SGLANG_SIMULATE_ACC_' | sort || true + echo "============================================================================" +} | tee "$SERVER_LOG" + +echo "Starting SGLang server for B300..." +"${SGLANG_CMD[@]}" >> "$SERVER_LOG" 2>&1 & +SERVER_PID=$! +echo "Server PID: $SERVER_PID" + +wait_for_server_ready --port "$SGLANG_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" + +if [ "$USE_SGLANG_ROUTER" = "true" ]; then + echo "Starting SGLang router on port $PORT for $TP DP ranks..." + python3 -m sglang_router.launch_router \ + --worker-urls "http://localhost:$SGLANG_BACKEND_PORT" \ + --policy consistent_hashing \ + --request-id-headers x-correlation-id \ + --dp-aware \ + --host 0.0.0.0 \ + --port "$PORT" \ + --prometheus-host 127.0.0.1 \ + --prometheus-port "$SGLANG_ROUTER_METRICS_PORT" \ + --connect-timeout-secs 900 \ + --request-timeout-secs 14400 \ + --disable-health-check \ + --disable-retries > "$ROUTER_LOG" 2>&1 & + ROUTER_PID=$! + echo "Router PID: $ROUTER_PID" + wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID" +fi + +if [ "${EVAL_ONLY}" = "true" ]; then + # GLM-5.2's chat template defaults to reasoning_effort=Max when the + # client passes no chat_template_kwargs (mini-swe-agent doesn't), and the + # heavy thinking burns the default 75-step budget: on the 23-instance + # slice, 12/23 trajectories exited LimitsExceeded unsubmitted while 10 of + # the 11 that submitted resolved. Double the step budget for this recipe; + # other recipes keep the shared 75 default. + export SWEBENCH_AGENT_STEP_LIMIT=150 + run_eval --port "$PORT" +else + build_replay_cmd "$RESULT_DIR" + REPLAY_CMD+=" --server-metrics http://localhost:$SGLANG_BACKEND_PORT/metrics" + run_agentic_replay_and_write_outputs "$RESULT_DIR" +fi diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index a30af59641..3787ef6fde 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -9053,6 +9053,39 @@ glm5.2-fp4-b300-sglang-agentic: # strictly dominated by conc 48 on both throughput and interactivity. - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48], router: { name: sglang-router, version: "0.3.2" } } +# GLM-5.2 B300 NVFP4 AgentX with EAGLE/MTP speculative decoding — the arm that +# replaces the STP entry above once agentic non-MTP is deprecated (MODELS.md, +# 2026-08-03). SGLang EAGLE off GLM-5.2's built-in nextn head (num-steps 3, +# eagle-topk 1, 4 draft tokens = 3 speculative tokens), with acceptance pinned +# to the golden AL 2.99 (golden_al_distribution/glm5.2_mtp.yaml, thinking_on, +# K=3) through SGLANG_SIMULATE_ACC_*. Image is bumped to v0.5.16-cu130: it is +# the first release that reads SGLANG_SIMULATE_ACC_TOKEN_MODE, and it is the +# tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run. +# +# Cookbook low-latency arm (TP8, fp8 KV, cutedsl bf16 GEMM) only, on the AgentX +# MTP concurrency grid [1, 4, 8, 12, 16]: steps of at least 2 because +# single-step sampling cannot separate configurations by more than run-to-run +# noise on the agentic corpus, and a hard stop at conc 16. The STP entry's DEP +# throughput arm is deliberately not carried over — its frontier peak is conc +# 48, well above that cap, and at conc <= 16 attention-DP leaves 2 sessions per +# rank and is strictly dominated by TP8, so it would spend a GPU job per point +# to re-measure a worse curve. Both arms of the STP entry offload to host DRAM, +# so this one does too: without HiCache the working set outgrows HBM past conc +# 8 and the radix hit rate collapses to <0.1. +glm5.2-fp4-b300-sglang-agentic-mtp: + image: lmsysorg/sglang:v0.5.16-cu130 + model: nvidia/GLM-5.2-NVFP4 + model-prefix: glm5.2 + runner: cluster:b300-nv + precision: fp4 + framework: sglang + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 4, 8, 12, 16] } + qwen3.5-fp8-gb200-dynamo-sglang-mtp: image: lmsysorg/sglang:v0.5.14-cu130@sha256:5027e95bf6ec536856b1b52a91d1f35ff5c564ab83e8a94758a169ff09bb8df3 model: Qwen/Qwen3.5-397B-A17B-FP8 diff --git a/golden_al_distribution/README.md b/golden_al_distribution/README.md index 93666c4fba..15dcb6d172 100644 --- a/golden_al_distribution/README.md +++ b/golden_al_distribution/README.md @@ -112,6 +112,7 @@ Before accepting an updated curve, reviewers should verify: | Qwen3.5 397B-A17B | MTP | [`qwen3.5_mtp.yaml`](qwen3.5_mtp.yaml) | [27317114007](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27317114007) | | Kimi K2.5 | EAGLE3 | [`kimik2.5_eagle3.yaml`](kimik2.5_eagle3.yaml) | [28122195822](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28122195822) | | MiniMax-M3 | EAGLE3 | [`minimaxm3_eagle3.yaml`](minimaxm3_eagle3.yaml) | [28061204145](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28061204145) | +| GLM-5.2 | MTP | [`glm5.2_mtp.yaml`](glm5.2_mtp.yaml) | [28058352479](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28058352479) | ## Primary references diff --git a/golden_al_distribution/README_zh.md b/golden_al_distribution/README_zh.md index e4231027e5..1285b613bd 100644 --- a/golden_al_distribution/README_zh.md +++ b/golden_al_distribution/README_zh.md @@ -112,6 +112,7 @@ gh workflow run speedbench-al.yml \ | Qwen3.5 397B-A17B | MTP | [`qwen3.5_mtp.yaml`](qwen3.5_mtp.yaml) | [27317114007](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27317114007) | | Kimi K2.5 | EAGLE3 | [`kimik2.5_eagle3.yaml`](kimik2.5_eagle3.yaml) | [28122195822](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28122195822) | | MiniMax-M3 | EAGLE3 | [`minimaxm3_eagle3.yaml`](minimaxm3_eagle3.yaml) | [28061204145](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28061204145) | +| GLM-5.2 | MTP | [`glm5.2_mtp.yaml`](glm5.2_mtp.yaml) | [28058352479](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28058352479) | ## 主要参考资料 diff --git a/golden_al_distribution/glm5.2_mtp.yaml b/golden_al_distribution/glm5.2_mtp.yaml new file mode 100644 index 0000000000..afebacac98 --- /dev/null +++ b/golden_al_distribution/glm5.2_mtp.yaml @@ -0,0 +1,19 @@ +# Source GitHub Actions run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28058352479 +# Acceptance Length (AL) reference values measured with SPEED-Bench. +# dataset: coding | temperature: 1.0 | top_p: 0.95 | output_len: 4096 +# thinking_on chat_template_kwargs: {"enable_thinking": true} +# thinking_off chat_template_kwargs: {"enable_thinking": false} +# Measured on glm-5.2-fp8 (B300, vLLM MTP), per num_speculative_tokens. +# Auto-generated by benchmarks/single_node/speedbench/glm52_fp4_b300_vllm.sh (speedbench-al.yml). +# +# key = num_speculative_tokens (MTP level); value = golden AL +glm-5.2-fp8: + thinking_on: + 1: 1.84 + 2: 2.50 + 3: 2.99 + 4: 3.33 + 5: 3.61 + 6: 3.78 + 7: 3.91 + 8: 4.06 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 6eebd97070..ccf6558de5 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5355,3 +5355,15 @@ - "Apply the accuracy-gated Kimi-K2.5 MXFP4 settings: tuned AITER MXFP4 MoE, fused shared experts, FP8 KV cache, block size 16, 16384 batched tokens, 512 sequences, async scheduling, gpu-memory-utilization 0.85 (headroom for CUDA-graph capture on MI355X), and the AITER BF16 GEMM path" - "Extend the TP4 and TP8 8k1k concurrency sweep from 64 to 128 (1k1k deprecated per #2263)" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2213 + +- config-keys: + - glm5.2-fp4-b300-sglang-agentic-mtp + description: + - "Add the GLM-5.2 NVFP4 B300 AgentX (agentic-coding) recipe with EAGLE/MTP speculative decoding, routed to benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang_mtp.sh. This is the arm that replaces glm5.2-fp4-b300-sglang-agentic once agentic non-MTP is deprecated for GLM-5.2 on 2026-08-03 (MODELS.md); the STP entry is left untouched here so the two can be compared before the cut-over." + - "Speculative config is GLM-5.2's built-in nextn head via SGLang EAGLE (no external draft model): --speculative-algorithm EAGLE, --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4, i.e. 3 speculative tokens per verification step. Same shape as the GLM-5.2 GB300 dynamo-sglang agentic recipes." + - "Throughput runs pin SGLang simulated acceptance to the committed golden AL: SGLANG_SIMULATE_ACC_LEN=2.99 (GLM-5.2 MTP curve at K=3, thinking_on), SGLANG_SIMULATE_ACC_METHOD=match-expected, SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token -- the triple golden_al_distribution/README.md prescribes for SGLang. EVAL_ONLY runs leave simulated acceptance off and keep real verification, since simulated acceptance commits drafted tokens regardless of the target logits and would score the eval at ~0." + - "Commits that curve as golden_al_distribution/glm5.2_mtp.yaml (SPEED-Bench coding, output-len 4096, temperature 1.0, top_p 0.95, thinking_on, num_speculative_tokens 1-8), verbatim from the speedbench-al.yml artifact of run 28058352479, and adds the GLM-5.2 row to golden_al_distribution/README.md and README_zh.md. The curve was collected on the FP8 checkpoint; one curve is committed per model, and the NVFP4 checkpoint ships the same nextn head." + - "Image bumped from lmsysorg/sglang:v0.5.15.post1-cu130 (the STP entry) to lmsysorg/sglang:v0.5.16-cu130. SGLANG_SIMULATE_ACC_TOKEN_MODE is only read from v0.5.16 -- SIMULATE_ACC_LEN / SIMULATE_ACC_METHOD exist further back, so an older image would silently ignore the token-mode half of the AgentX contract -- and v0.5.16-cu130 is the tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run." + - "Serve shape is the validated STP script unchanged outside the speculative and simulated-acceptance blocks: cookbook B300 NVFP4 low-latency arm (TP8, --kv-cache-dtype fp8_e4m3, --bf16-gemm-backend cutedsl, --max-prefill-tokens 8192, --chunked-prefill-size 8192, --mem-fraction-static 0.85, glm47 tool-call parser, glm45 reasoning parser) with HiCache host-DRAM offload at hicache-ratio 0.75 / write_back / direct / page_first_direct. mem-fraction-static stays at 0.85, which is also what the GLM-5.2 GB300 aggregated MTP recipe uses; the hicache ratio is relative to the device pool, so the nextn layer's own KV only shrinks the host pool." + - "Search space is one arm on the AgentX MTP concurrency grid: TP8 + HiCache at conc [1, 4, 8, 12, 16]. Steps of at least 2 because single-step sampling cannot separate configurations by more than run-to-run noise on the agentic corpus, and a hard stop at conc 16. The STP entry's DEP (attention-DP + EP8) throughput arm is not carried over: its frontier peak is conc 48, above that cap, and at conc <= 16 attention-DP leaves 2 sessions per rank and is strictly dominated by TP8, so it would spend one GPU job per point re-measuring a worse curve. The DEP branch is kept in the benchmark script -- including the --speculative-moe-a2a-backend none / --speculative-moe-runner-backend triton pair that GLM-5.2-NVFP4's unquantized bf16 nextn layer needs once expert parallelism puts an all-to-all in the MoE path -- so the throughput arm can be added later without re-deriving it." + pr-link: TBD From a3c7a4c5e1d01e10bf55bd8823ddaac4f1ec538f Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 1 Aug 2026 22:44:28 -0400 Subject: [PATCH 2/2] chore: fill perf-changelog pr-link for #2447 Co-Authored-By: Claude Opus 5 (1M context) --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index ccf6558de5..069a76f349 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5366,4 +5366,4 @@ - "Image bumped from lmsysorg/sglang:v0.5.15.post1-cu130 (the STP entry) to lmsysorg/sglang:v0.5.16-cu130. SGLANG_SIMULATE_ACC_TOKEN_MODE is only read from v0.5.16 -- SIMULATE_ACC_LEN / SIMULATE_ACC_METHOD exist further back, so an older image would silently ignore the token-mode half of the AgentX contract -- and v0.5.16-cu130 is the tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run." - "Serve shape is the validated STP script unchanged outside the speculative and simulated-acceptance blocks: cookbook B300 NVFP4 low-latency arm (TP8, --kv-cache-dtype fp8_e4m3, --bf16-gemm-backend cutedsl, --max-prefill-tokens 8192, --chunked-prefill-size 8192, --mem-fraction-static 0.85, glm47 tool-call parser, glm45 reasoning parser) with HiCache host-DRAM offload at hicache-ratio 0.75 / write_back / direct / page_first_direct. mem-fraction-static stays at 0.85, which is also what the GLM-5.2 GB300 aggregated MTP recipe uses; the hicache ratio is relative to the device pool, so the nextn layer's own KV only shrinks the host pool." - "Search space is one arm on the AgentX MTP concurrency grid: TP8 + HiCache at conc [1, 4, 8, 12, 16]. Steps of at least 2 because single-step sampling cannot separate configurations by more than run-to-run noise on the agentic corpus, and a hard stop at conc 16. The STP entry's DEP (attention-DP + EP8) throughput arm is not carried over: its frontier peak is conc 48, above that cap, and at conc <= 16 attention-DP leaves 2 sessions per rank and is strictly dominated by TP8, so it would spend one GPU job per point re-measuring a worse curve. The DEP branch is kept in the benchmark script -- including the --speculative-moe-a2a-backend none / --speculative-moe-runner-backend triton pair that GLM-5.2-NVFP4's unquantized bf16 nextn layer needs once expert parallelism puts an all-to-all in the MoE path -- so the throughput arm can be added later without re-deriving it." - pr-link: TBD + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2447