feat(pytorch): add glm5.3-flash support by qescccczmr · Pull Request #4968 · InternLM/lmdeploy

qescccczmr · 2026-09-14T13:18:23Z

Summary

Add GLM-5.3 Flash text, image and video support to the PyTorch engine, including MTP speculative decoding, tensor parallelism and combined data/expert parallelism.

Implemented features

Load the FP8 checkpoint with existing model loaders and MoE components; integrate hybrid KDA/sparse MLA attention, KPool state management and the compact 512-value NoPE cache.
Reuse FLA and shared recurrent/convolution kernels, FlashMLA, and common vision LayerNorm/RoPE. Preserve FP32 recurrent states and MoE/TP reductions, routed-expert scaling, and BF16 rounding boundaries.
Support MTP with per-token state checkpoints and target-to-draft multimodal embedding handoff; support TP, DP/EP, decode CUDA Graphs and optional prefill CUDA Graphs.
Provide multimodal processor compatibility fallback for older Transformers releases.

Optimizations

Fuse KPool verification/checkpoint updates, skip compression for unclosed pools, bound prefill scoring workspace, and broadcast selected groups before local expansion.
Consume strided KDA inputs and merge/batch projections, reducing projection launches 6 → 4 while preserving checkpoint names, TP shards and LoRA fallback.
Fuse mHC reduction/RMSNorm and FP32 preparation, sparse-index preparation, and valid-row MoE activation/FP8 quantization; reuse MTP key/score projections.

LMDeploy serving results

H200, GLM-5.3 Flash FP8 weights, BF16 KV, FP32 recurrent state, prefix caching off, greedy/seed 42, fixed output lengths. Timings exclude active profiling. These measurements belong to the revisions below, not a fresh benchmark of current head a5456e2b.

TP4 / EP1 / DP1

Measured at ec2ec448: BS1, MTP5, 5 output tokens, max batch 16, prefill chunk 2,048, session length 270,336; 3 warmups + 30 measured requests per input. Values are independently computed medians.

Input tokens	TTFT (ms)	TPOT (ms)	Total request (ms)
2,048	234.60	7.28	264.15
8,192	892.82	7.08	921.83

The 8K input uses four prefill chunks. TPOT includes the final SSE finish response; these short-output request timings are not single-forward GPU latency.

TP1 / DP4 / EP4

Measured at a1a7955c on 2026-10-08: 2,048 input / 1,024 output / global concurrency 16, max batch 16 per rank, prefill chunk 2,048, session length 16,384. Per K: 16 warmups, then 3 rounds × 64 measured requests. Other workloads occupied the remaining four GPUs.

K	Output tok/s	Mean TTFT (ms)	Mean TPOT (ms)	MTP acceptance	Mean accepted length, including bonus
1	1024.09	398.85	15.059	95.38%	1.954
3	1293.12	375.08	11.069	83.06%	3.492
5	1362.46	374.15	9.986	71.54%	4.577

TP1 / DP8 / EP8

Measured at 71797d80 on 2026-10-09: 2,048 input / 1,024 output / global concurrency 32, four client workers per rank, max batch 32 per rank, prefill chunk 2,048, session length 8,192. Fresh service per K; 64 warmups + 256/256 successful measured requests per run. DP/EP runs use DeepEP Normal for globally mixed/prefill steps, Low Latency for global decode, and prefill CUDA Graphs up to 2,048 tokens.

K	Output tok/s	Mean TTFT (ms)	Mean TPOT (ms)	MTP acceptance	Mean accepted length, including bonus
1	1763.37	412.05	17.031	94.34%	1.943
3	2292.07	318.60	12.609	80.46%	3.414
5	2425.87	341.65	10.868	68.88%	4.444
1 (repeat)	1848.06	326.38	16.051	94.24%	1.942

K=1 showed run-to-run variation; both runs are retained. The repeat used the same commit/protocol from a local checkout, while the first used a shared-storage checkout; this does not isolate the cause of the variation.

K counts draft tokens; verification can contain K+1 tokens. TPOT = (last SSE response − first SSE response)/(output tokens − 1). DP/EP values average requests; throughput divides generated tokens by measured wall time. Acceptance = accepted/proposed draft tokens; accepted length = 1 + accepted/draft iterations. MTP can emit multiple tokens per SSE event.

Accuracy

Historical LMDeploy evaluations using FP8 weights, TP4/MTP5 and NCCL. These are not fresh accuracy evaluations of the latest head.

Benchmark	Correct / total	Score
GSM8K test	1,290 / 1,319	97.8014%
MMMU-Pro vision/test	1,333 / 1,730	77.0520%
GPQA Diamond	184 / 198	92.9293%
SciCode with background: subproblems	151 / 288	52.4306%
SciCode with background: main problems	13 / 65	20.0000%

GSM8K: 2026-09-24 frozen working-tree snapshot (8068cd43 plus recorded patches); greedy, concurrency 32, 2,048-token output cap. All 1,319 items remain in the denominator, including six length-stopped responses; explicit final answers are scored when present.
MMMU-Pro: 2026-09-24 frozen snapshot (3f21b23f plus evaluated patches, later published as 38779f12). Includes one same-budget, fresh-service retry selected by length stop; final aggregate has zero request errors or remaining length stops.
GPQA/SciCode: measured at 38779f12, temperature 1.0, top_p 0.95, seed 42, one sample per item. SciCode covers all 65 test problems/288 scored subproblems, excluding three official prefilled steps. Scores include the published-helper correction and length-selected retries with dependent-step regeneration; assertions and tolerances are unchanged.

Accuracy generation budgets

MMMU-Pro uses a 327,680-token output cap. GPQA/SciCode start with the same cap; length-stopped items are retried once at 655,360 tokens, including dependent SciCode steps. All selected replacements are retained regardless of correctness; incorrect answers alone do not trigger retries.

Local regression checks cover processor compatibility, MoE option propagation and communication fallback; the latest communicator check passed all 7 tests, and changed-file lint passed.

Reuse MLA/MoE loaders, FA3, mHC, sparse Top-K and compact DeepGEMM. Add KDA state adaptation and multimodal GLM configuration/processing. Preserve existing defaults and the positional MoE prefix argument.

yidingcheng0206 · 2026-09-21T03:25:25Z

+            device=device,
+            is_tp=True,
+            quant_config=None,
+            dp_disable_tp=True,


Keep KV-B sharding consistent with the attention TP group
When dp > 1 and attn_tp > 1, dp_disable_tp=True makes kv_b_proj retain all attention heads, while DeepseekV2BMM still shards kc/vc by attn_tp. The inherited process_weights_after_loading() then copies the full KV-B-derived weights directly into these sharded tensors.

Fixed in 3fcabb7. Removed dp_disable_tp=True from GLM's KV-B projection so it uses the existing attention-TP sharding and weight loader, consistent with DeepseekV2BMM KC/VC absorption.

The focused CPU test exercises the real builder/loader/absorption in nine DP/TP/rank combinations and checks the exact local weight slices. This is loader-contract validation, not an EP+DP distributed-service validation; the unrelated EP+DP experiments are not included.

RunningLeon · 2026-09-21T05:59:29Z

@qescccczmr Hi, is this PR ready to review?

RunningLeon · 2026-09-21T07:22:13Z

+from .step_metadata import register_step_metadata_impl
+
+
+def _select_state(state: torch.Tensor, metadata: Any) -> torch.Tensor:


can we use existing _state_select , _state_scatter in here?

lmdeploy/lmdeploy/pytorch/backends/cuda/gated_delta_rule.py

Line 201 in 9adf485

def _state_select(state, state_indices, spec_offsets):

lmdeploy/lmdeploy/pytorch/backends/cuda/gated_delta_rule.py

Line 232 in 9adf485

def _state_scatter(state, state_indices, spec_offsets, src):

Addressed in 3fcabb7. KDA now reuses the existing GDN _state_select / _state_scatter helpers for both AR and MTP. The ordinary AR bank is exposed as a one-slot ring, rather than introducing another state kernel.

The shared helpers mask invalid state/slot IDs: reads return zero and writes are skipped, preventing padded rows from overwriting a live request. Actual CUDA tests cover FP32/BF16, irregular shapes, negative/out-of-range IDs, initialization, dummy/live-row collisions, and replay after updating graph inputs.

RunningLeon · 2026-09-21T07:23:37Z

+        head_dim: int,
+        lower_bound: float,
+    ) -> torch.Tensor:
+        if (metadata.spec_state_offsets is not None


mtp may need support as well

Updated in 3fcabb7. KDA restores state using accepted history and saves every verified token's convolution/recurrent checkpoint, including partial rejection, reordering and dummy rows. MTP now also honors index_share_for_mtp_iteration through the existing DSATopKIndicesBuffer; cache updates still run on every draft step. Growing the shared buffer changes the graph key so old allocations are not replayed.

We also found and fixed a sampling-distribution bug: rejection/recovery did not apply AR's top-k/top-p/min-p filters. The fix reuses FusedLogitsProcessor filtering, preserves greedy fast-path behavior, and leaves source logits intact. Focused CUDA tests verify that filtered-out tokens cannot be accepted or recovered.

Mode LMDeploy vLLM

Greedy, fixed 256 tokens 62.9341% 62.7595%

Sampling, fixed 256 tokens 54.7305% 53.5599%

Greedy, natural EOS (max 256) 62.4700% 61.5550%

Sampling, natural EOS (max 256) 56.9082% 57.5145%

H200, TP4/EP1, CUDA Graph on, prefix cache off, BF16 KV, BS1, 1024-token prefill chunks. Inputs: 20 GSM8K prompts plus four 8436-token long prompts, each run under all four policies; sampling uses temperature=1, top_p=0.95, seed=42 and disabled top-k. Stop IDs/minimum lengths are matched.

Reuse GDN state helpers and channelwise TileLang verification; honor MTP index sharing and invalidate graphs after buffer growth. Share AR sampling filters with rejection/recovery and stabilize GLM TP/mHC and sparse-index arithmetic. Keep EP/DP, prefix-cache and performance experiments outside this patch.

qescccczmr · 2026-09-22T07:30:02Z

@RunningLeon The MTP correctness/reuse changes are ready for another review in 3fcabb7. I am keeping this PR in Draft while whole-model accuracy/performance validation remains incomplete.

This update modifies 11 existing production files, with no new production module/kernel. It addresses the four review comments: attention-TP KV-B sharding, shared state helpers, accepted-history MTP state recovery, and reuse of GDN's TileLang verification kernel. It also fixes rejection sampling's missing AR filters, MTP index sharing and graph-buffer growth, and includes the GLM opt-in FP32 TP / mHC / stable sparse-index changes used for AR/MTP numerical stability. Existing defaults for other model callers are preserved. EP+DP, prefix-cache, FlashInfer communication and compact-MoE scheduling experiments are excluded.

RunningLeon · 2026-09-22T09:51:59Z

    prefix: str = '',
+    *,
+    fp32_acc: bool = False,
+    output_scale: float = 1.0,


[P1] Honor or explicitly reject the new MoE options across backend branches
build_fused_moe() now exposes fp32_acc and output_scale, but only the blocked-FP8 branch forwards them. The BF16 and W8A8 branches silently ignore these arguments, so the same public API has different numerical semantics depending on the selected backend.
This causes a concrete correctness issue for GLM’s BF16 path: the router returns unscaled weights and relies on output_scale=2.5 in the expert reduction. Dropping that argument produces an incorrectly scaled routed-expert contribution.
Please propagate these options through supported implementations, preserving the FP32 weighted reduction → output scaling → output cast order. For implementations that do not support this contract yet, explicitly reject non-default values rather than silently ignoring them. Keep False / 1.0 as defaults to preserve existing behavior, and add regression tests for parameter propagation, numerical correctness, and unsupported-backend rejection.
The scale must apply only to the routed-expert contribution, not to the shared-expert output.

Fixed in 96423321.

The BF16 fused-MoE path now forwards fp32_acc and output_scale through its build spec and applies them in the CUDA reduction as FP32 weighted reduction → scale → output cast.

The scale is applied inside the routed-expert reduction, before the caller combines the result with the shared expert output.

W8A8, static-FP8, W4A16, and DeepEP branches now explicitly reject non-default values instead of silently ignoring them.

Added propagation, unsupported-backend, and CUDA numerical regression tests in tests/pytorch/nn/test_moe_options.py.

Validation: pytest -q tests/pytorch/nn/test_moe_options.py (4 passed, 1 CUDA test skipped because CUDA is unavailable in this environment); Ruff and compile checks pass.

Reject unsupported DLINFER MoE reduction options instead of ignoring them. Leave NCCL NVLS policy to the runtime and remove model-specific environment plumbing. Document GLM FP32 operator usage and cover dtype, reduction, and environment contracts. Validation: 110 focused tests passed on CPU/H200; Ruff passed for lmdeploy and tests/pytorch. DLINFER rejection regressions failed before the fix and pass afterward.

Apply the configured string and docstring formatters to GLM PR files. Verified executable AST is unchanged; both previously failing hooks and Ruff pass locally.

Batch KDA gate projections through the linear backend while preserving checkpoint loaders, TP shards and the original LoRA path. Reuse MTP key/score projections, fuse raw token-cache indexing, and share target KPool metadata within each eager or captured forward. Prepare the next mHC FP32 input in post-expand after the original BF16 rounding boundary. Keep FP32 GEMM, normalization and TP reductions intact. Validation: full-model 2K/8K token and logits parity; ragged/rejection cache replay; TP1/4/8 and DP4/EP4 loading; real LoRA TP4 checks; existing HC tests; paired serving A/B and separate four-rank Torch Profiler.

Keep renormalization unscaled; normal FP32 gather and low-latency combine own the single output_scale factor. Add a cross-layer regression that fails with 6.25 instead of 2.5 before this fix. AI-assisted local campaign fix. No push.

Merge 54d2665 while preserving PR2 raw-ring ownership, compact KPool pages, and single routed-expert scaling. Adapt shared decode metadata and bounded prefill scoring to both page layouts and DeepGEMM APIs. Retain upstream FP8 activation fusion and mHC/prefill graph changes. Validation: 306 targeted CPU/CUDA tests passed on GPU7, changed-file Ruff and diff checks passed. No model weights loaded; full distributed service and GSM8K acceptance remain pending.

Copilot

🔵 Needs a closer look

Unresolved processor fallback and untested speculative-state rollback accompany extensive CUDA and distributed changes.

3 open findings

Fallback processor is unreachable on older Transformers releases · New
Add rollback tests for speculative KDA circular checkpoints · New
Test top-p and min-p filtering in rejection sampling · New

What changed in this PR

Adds GLM-5.3 Flash multimodal inference, MTP, hybrid KDA/KPool attention, and distributed execution support to the PyTorch engine.

Changes:

Implements GLM-5.3 configuration, model, vision, KDA, KPool, and cache handling.
Extends CUDA kernels and MoE/communication backends for FP32 accumulation and optimized execution.
Aligns speculative sampling filters and MTP state handling with autoregressive generation.

File	Description
`requirements/runtime_cuda.txt`	Pins the required FLA version.
`lmdeploy/vl/model/glm5_next.py`	Adds multimodal preprocessing.
`lmdeploy/vl/model/builder.py`	Registers the vision frontend.
`lmdeploy/pytorch/models/glm5_next.py`	Implements the GLM-5.3 model.
`lmdeploy/pytorch/third_party/deep_gemm/__init__.py`	Adds DeepGEMM API compatibility.
`lmdeploy/pytorch/spec_decode/reject_sampler.py`	Filters speculative target distributions.
`lmdeploy/pytorch/spec_decode/proposers/deepseek_mtp.py`	Shares draft embeddings.
`lmdeploy/pytorch/nn/rotary_embedding.py`	Adds FP32 RoPE computation.
`lmdeploy/pytorch/nn/moe/default.py`	Propagates MoE precision options.
`lmdeploy/pytorch/nn/moe/blocked_fp8.py`	Extends FP8 MoE behavior.
`lmdeploy/pytorch/nn/moe/__init__.py`	Exposes MoE scaling options.
`lmdeploy/pytorch/nn/linear/default.py`	Adds batched projections.
`lmdeploy/pytorch/nn/linear/base.py`	Supports promoted TP reductions.
`lmdeploy/pytorch/nn/linear/__init__.py`	Exposes batched linear construction.
`lmdeploy/pytorch/nn/kda.py`	Adds backend-dispatched KDA.
`lmdeploy/pytorch/nn/kpool.py`	Defines reusable KPool semantics.
`lmdeploy/pytorch/nn/hc_prepost.py`	Extends mHC fused paths.
`lmdeploy/pytorch/nn/attention.py`	Exposes latent-cache flattening.
`lmdeploy/pytorch/nn/__init__.py`	Exports KDA and KPool modules.
`lmdeploy/pytorch/models/module_map.py`	Registers GLM-5.3 architectures.
`lmdeploy/pytorch/models/glm4_1v.py`	Refines vision RoPE initialization.
`lmdeploy/pytorch/models/glm_moe_dsa_mtp.py`	Generalizes MTP decoder construction.
`lmdeploy/pytorch/models/deepseek_v32.py`	Makes sparse MLA selectable.
`lmdeploy/pytorch/models/deepseek_v2.py`	Generalizes routed MoE scaling.
`lmdeploy/pytorch/kernels/cuda/sparse_index_topk.py`	Adds deterministic strided Top-K.
`lmdeploy/pytorch/kernels/cuda/moe/route_noaux_tc.py`	Supports 288-expert routing.
`lmdeploy/pytorch/kernels/cuda/moe/fused_moe.py`	Adds FP32 reduction and scaling.
`lmdeploy/pytorch/kernels/cuda/moe/ep.py`	Supports FP32 EP gathering.
`lmdeploy/pytorch/kernels/cuda/moe/ep_fp8.py`	Extends FP8 expert execution.
`lmdeploy/pytorch/kernels/cuda/moe/blocked_fp8.py`	Tunes blocked-FP8 MoE.
`lmdeploy/pytorch/kernels/cuda/kpool.py`	Implements fused KPool kernels.
`lmdeploy/pytorch/kernels/cuda/gated_delta_rule.py`	Adds channelwise KDA recurrence.
`lmdeploy/pytorch/kernels/cuda/flatten_kv_cache.py`	Expands shared-cache handling.
`lmdeploy/pytorch/kernels/cuda/fill_kv_cache.py`	Adds indexed cache writes.
`lmdeploy/pytorch/kernels/cuda/dsv4/hc_prepost.py`	Fuses mHC normalization/preparation.
`lmdeploy/pytorch/kernels/cuda/causal_conv1d.py`	Supports strided mixed-dtype inputs.
`lmdeploy/pytorch/kernels/cuda/apply_rotary_pos_emb.py`	Implements FP32 RoPE kernels.
`lmdeploy/pytorch/kernels/cuda/activation.py`	Fuses clamped SwiGLU quantization.
`lmdeploy/pytorch/engine/logits_process.py`	Shares sampling-filter logic.
`lmdeploy/pytorch/engine/executor/base.py`	Reserves workspace and rejects prefix caching.
`lmdeploy/pytorch/consts.py`	Defines named GLM state caches.
`lmdeploy/pytorch/configurations/glm5_next.py`	Builds GLM runtime/cache configuration.
`lmdeploy/pytorch/config.py`	Adds model capability flags.
`lmdeploy/pytorch/backends/moe.py`	Extends MoE build specifications.
`lmdeploy/pytorch/backends/linear.py`	Adds batched-linear interface.
`lmdeploy/pytorch/backends/kda.py`	Defines the KDA backend contract.
`lmdeploy/pytorch/backends/hc_prepost.py`	Extends the mHC backend contract.
`lmdeploy/pytorch/backends/dlinfer/op_backend.py`	Rejects unsupported FP32 RoPE.
`lmdeploy/pytorch/backends/dlinfer/moe.py`	Rejects unsupported MoE options.
`lmdeploy/pytorch/backends/default/op_backend.py`	Configures default FP32 RoPE.
`lmdeploy/pytorch/backends/default/linear.py`	Implements batched projections.
`lmdeploy/pytorch/backends/default/apply_rotary_emb.py`	Implements default FP32 RoPE.
`lmdeploy/pytorch/backends/cuda/op_backend.py`	Registers CUDA KDA and RoPE.
`lmdeploy/pytorch/backends/cuda/nsa.py`	Bounds aligned score workspaces.
`lmdeploy/pytorch/backends/cuda/moe/default.py`	Propagates CUDA MoE precision.
`lmdeploy/pytorch/backends/cuda/moe/blocked_fp8.py`	Extends DeepEP FP8 execution.
`lmdeploy/pytorch/backends/cuda/kpool.py`	Adapts KPool to CUDA caches.
`lmdeploy/pytorch/backends/cuda/kda.py`	Implements CUDA KDA state handling.
`lmdeploy/pytorch/backends/cuda/hc_prepost.py`	Connects fused mHC kernels.
`lmdeploy/pytorch/backends/cuda/gated_delta_rule.py`	Hardens indexed state operations.
`lmdeploy/pytorch/backends/cuda/comm/flashinfer_allreduce.py`	Adds FP32 and multi-workspace support.
`lmdeploy/pytorch/backends/cuda/attention/sparse_mla.py`	Fuses sparse-index remapping.
`lmdeploy/pytorch/backends/cuda/attention/mla.py`	Adds latent-cache fill/flatten.
`lmdeploy/pytorch/backends/cuda/apply_rotary_emb.py`	Enables CUDA FP32 RoPE.
`lmdeploy/pytorch/backends/attention.py`	Defines latent-cache API.
`lmdeploy/pytorch/backends/apply_rotary_emb.py`	Adds the FP32 build option.
`lmdeploy/hf_configs/configuration_glm5_next.py`	Adds local Transformers configuration.
`lmdeploy/hf_configs/__init__.py`	Registers the local configuration.
`lmdeploy/archs.py`	Recognizes GLM-5.3 as multimodal.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

+        processor = AutoProcessor.from_pretrained(
+            self.model_path, trust_remote_code=trust_remote_code)
+        if not (hasattr(processor, 'image_processor')
+                and hasattr(processor, 'video_processor')):
+            processor = self._build_compat_processor(trust_remote_code)


+        output, _ = self._decode_recurrent(
+            q, k, v, g=raw_gate.reshape(batch, steps, heads, dim).contiguous(),
+            beta=raw_beta.reshape(batch, steps, heads).contiguous(),
+            A_log=kwargs['a_log'], dt_bias=kwargs['dt_bias'], lower_bound=kwargs['lower_bound'],
+            initial_state=recurrent_state, state_indices=signed_ids, cache_seqlens=history,
+            output_final_state=True)


+        # Verification/recovery must use the AR sampling distribution too.
+        # Filter out of place so bonus sampling and returned logits are intact.
+        filtered_logits = FusedLogitsProcessor(
+            expanded_sampling_inputs).filter_logits(target_logits.flatten(0, 1))
+        target_draft_logits = filtered_logits.view_as(target_logits)[:, :-1].contiguous()


RunningLeon · 2026-10-08T06:32:34Z

@qescccczmr ut has failed as :

    def fusedmoe_build(self, low_latency_mode: bool = False):
        deepep_moe = _build_deepep_moe(low_latency_mode,
                                       self.ep_size,
                                       self.ep_group,
                                       self.num_experts,
                                       self.hidden_dim,
                                       self.block_size,
                                       self.top_k,
                                       self.out_dtype,
                                       fp8_dtype=self.fp8_dtype,
                                       scale_fmt=self.scale_fmt,
                                       layer_idx=self.layer_idx,
                                       num_max_dispatch_tokens_per_rank=self.num_max_dispatch_tokens_per_rank,
>                                      fp32_acc=self.fp32_acc,
                                                ^^^^^^^^^^^^^
                                       chunk_size=16 * 1024)
E       AttributeError: 'FusedDeepEpMoEBlockedF8Impl' object has no attribute 'fp32_acc'

Handle unsupported AutoProcessor import/value errors with the existing GLM-4V compatibility builder. Cover native, missing-modality, error propagation and trust-remote-code paths. Update DeepEP FP32 option and FlashInfer dtype/workspace fixtures, and repair the two dead Llama 3.1 documentation links.

Fuse target raw-ring updates and restore fused draft decoding while preserving cache state semantics.

RunningLeon · 2026-10-09T02:07:54Z

-        processor = AutoProcessor.from_pretrained(
-            self.model_path, trust_remote_code=trust_remote_code)
+        try:
+            processor = AutoProcessor.from_pretrained(


since glm5.3 flash is a vl model, the processor set to None is not a good idea. let's check why it failed? is it because transformer's version? if that is the case, we can check it and raise error to let user know

Integrate PR2 with PR4968 development branch

RunningLeon · 2026-10-10T01:20:53Z

 def _filter_minp_sorted_(scores: torch.Tensor, minp: torch.Tensor, filter_value: float = -float('inf')):
    """Filter minp on sorted scores."""
-    softmax_scores = scores.softmax(-1)
+    softmax_scores = scores.softmax(-1, dtype=torch.float32)


why need to use fp32 explicitly for this model?

Preserve GLM KPool tail-index padding alongside upstream DCP topk_length and LSE handling. Add regression coverage for aligned and unaligned indices, TP head padding, and both LSE layouts. Validation: 60 existing CPU tests and 8 added regression cases passed; 27 GPU tests skipped because GPUs are occupied. Ruff, docformatter, changed-file syntax checks, and git diff --check passed.

RunningLeon · 2026-10-13T01:33:40Z

@qescccczmr can you fix ut?

qescccczmr added 2 commits September 14, 2026 12:43

feat(pytorch): support GLM-5.3 Flash with shared LMDeploy components

1361ba7

Reuse MLA/MoE loaders, FA3, mHC, sparse Top-K and compact DeepGEMM. Add KDA state adaptation and multimodal GLM configuration/processing. Preserve existing defaults and the positional MoE prefix argument.

merge: adapt GLM-5.3 Flash to current main interfaces

e2fc086

qescccczmr changed the title ~~Feat/glm5.3 flash lmdeploy reuse~~ Feat/glm5.3 flash Sep 14, 2026

qescccczmr changed the title ~~Feat/glm5.3 flash~~ feat(pytorch): add glm5.3-flash support Sep 14, 2026

lvhan028 mentioned this pull request Sep 15, 2026

[Feature] 现在lmdeploy能支持模型GLM-5.3-Flash-W4A16-MTP运行吗？ #4958

Open

qescccczmr added 3 commits September 17, 2026 05:18

refactor(pytorch): use default Triton MoE for GLM-5.3 Flash

fca7e9f

refactor(pytorch): remove unused EP1 DeepGEMM MoE extensions

d3dbefe

merge: resolve main conflicts and adapt GLM indexer prefix

6b183f6

yidingcheng0206 reviewed Sep 21, 2026

View reviewed changes

RunningLeon requested a review from yidingcheng0206 September 21, 2026 06:00

merge: sync GLM-5.3 Flash PR with latest main

3bc41ce

RunningLeon self-requested a review September 21, 2026 06:59

RunningLeon reviewed Sep 21, 2026

View reviewed changes

feat(pytorch): support GLM-5.3 Flash MTP with shared components

6218f59

RunningLeon reviewed Sep 21, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/backends/cuda/kda.py Outdated

RunningLeon reviewed Sep 22, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/configurations/glm5_next.py Outdated

RunningLeon reviewed Sep 22, 2026

View reviewed changes

qescccczmr added 2 commits September 22, 2026 09:56

refactor(pytorch): reuse causal convolution cache for GLM MTP

41563c7

fix(pytorch): honor MoE reduction options across backends

9642332

qescccczmr marked this pull request as ready for review September 22, 2026 11:19

RunningLeon reviewed Sep 23, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/nn/rotary_embedding.py Outdated

RunningLeon reviewed Sep 23, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/nn/norm.py Outdated

RunningLeon reviewed Sep 23, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/backends/moe.py Outdated

RunningLeon reviewed Sep 23, 2026

View reviewed changes

Comment thread lmdeploy/pytorch/config.py Outdated

qescccczmr added 2 commits September 23, 2026 09:53

style(pytorch): satisfy GLM pre-commit hooks

a17e90b

Apply the configured string and docstring formatters to GLM PR files. Verified executable AST is unchanged; both previously failing hooks and Ruff pass locally.

qescccczmr and others added 10 commits September 29, 2026 05:28

style: wrap MoE reduction docstring for lint

932285b

perf(pytorch): fuse masked GLM expert activation and FP8 quantization

e7be6ba

perf: preserve padded KPool score strides for top-k

ed1eea0

perf: extend mHC FP32 preparation to 32-token batches

209a755

perf: enable GLM prefill CUDA graphs without token padding

b266581

fix: widen expert activation offsets for large capacity

54d2665

refactor: defer GLM prefill piecewise CUDA graphs

37bce48

RunningLeon requested a balanced review from Copilot October 8, 2026 03:53

Copilot started reviewing on behalf of RunningLeon October 8, 2026 03:54 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

revert: restore GLM prefill piecewise CUDA graphs

a1a7955

yidingcheng0206 and others added 4 commits October 8, 2026 15:12

Remove integration-specific test changes from PR

bc04f66

Remove CPU fallbacks from CUDA raw-ring helpers

2a406fa

Optimize GLM-5.3 KPool decode cache updates

e5a8568

Fuse target raw-ring updates and restore fused draft decoding while preserving cache state semantics.

RunningLeon reviewed Oct 9, 2026

View reviewed changes

yidingcheng0206 and others added 5 commits October 9, 2026 10:21

Fix KPool decode scoring overhead from graph padding

ddc7769

refactor: remove unrelated docs and FlashInfer changes from GLM PR

a5456e2

Refactor GLM-5.3 KPool cache update paths

1e2d8c5

Merge pull request #3 from yidingcheng0206/feat/glm53-integrated

6573655

Integrate PR2 with PR4968 development branch

fix(pytorch): preserve FP32 probabilities in sampling

c7b4343

RunningLeon reviewed Oct 10, 2026

View reviewed changes

qescccczmr added 3 commits October 10, 2026 02:12

refactor(pytorch): keep min-p outside sampling alignment fix

208fab1

revert: remove FP32 softmax sampling changes

b8138bc

		from .step_metadata import register_step_metadata_impl


		def _select_state(state: torch.Tensor, metadata: Any) -> torch.Tensor:

Mode	LMDeploy	vLLM
Greedy, fixed 256 tokens	62.9341%	62.7595%
Sampling, fixed 256 tokens	54.7305%	53.5599%
Greedy, natural EOS (max 256)	62.4700%	61.5550%
Sampling, natural EOS (max 256)	56.9082%	57.5145%

Conversation

qescccczmr commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implemented features

Optimizations

LMDeploy serving results

TP4 / EP1 / DP1

TP1 / DP4 / EP4

TP1 / DP8 / EP8

Accuracy

Uh oh!

yidingcheng0206 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

qescccczmr Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

RunningLeon commented Sep 21, 2026

Uh oh!

RunningLeon Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

qescccczmr Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

RunningLeon Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

qescccczmr Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qescccczmr commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

RunningLeon Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qescccczmr Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

RunningLeon commented Oct 8, 2026

Uh oh!

RunningLeon Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RunningLeon Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

RunningLeon commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

qescccczmr commented Sep 14, 2026 •

edited

Loading

qescccczmr Sep 22, 2026 •

edited

Loading

qescccczmr commented Sep 22, 2026 •

edited

Loading

RunningLeon Sep 22, 2026 •

edited

Loading

RunningLeon Oct 9, 2026 •

edited

Loading