fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path - #176
Open
potoior wants to merge 1 commit into
Open
fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path#176potoior wants to merge 1 commit into
potoior wants to merge 1 commit into
Conversation
QUANT_NVFP4_A16 (nvfp4_a16) was introduced in 0.10.0 to distinguish W4A16 weight-only NVFP4 from W4A4 NVFP4, but Qwen3SparseMoeBlock._prepare_moe_weights only gated on QUANT_NVFP4. Mixed-precision W4A16_NVFP4 checkpoints such as Qwen3.6-35B-A3B-NVFP4 therefore fell through to the INT4 GPTQ Marlin repack and crashed with: AttributeError: 'NVFP4A16MarlinLinear' object has no attribute 'qweight' Two changes: - Treat QUANT_NVFP4_A16 as NVFP4 when gating the NVFP4 MoE plugin path. - Accept NVFP4A16MarlinLinear experts in repack_nvfp4_gated_moe_experts (their raw weight/weight_scale/weight_scale_2 buffers are the same ModelOpt NVFP4 format, so decode_modelopt_nvfp4 reuses them as-is). Verified: tensorrt-edgellm-export completes for Qwen3.6-35B-A3B-NVFP4 (llm).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QUANT_NVFP4_A16was introduced in 0.10.0 to distinguish W4A16 weight-onlyNVFP4 from W4A4 NVFP4, but
Qwen3SparseMoeBlock._prepare_moe_weightsonlygated on
QUANT_NVFP4. Mixed-precisionW4A16_NVFP4checkpoints such asQwen3.6-35B-A3B-NVFP4therefore fell through to the INT4 GPTQ Marlinrepack
and crashed with:
Changes
modeling_qwen3_moe.py: treatQUANT_NVFP4_A16as NVFP4 when gating theNVFP4 MoE plugin path.
repacking.py: acceptNVFP4A16MarlinLinearexperts inrepack_nvfp4_gated_moe_experts(their rawweight/weight_scale/weight_scale_2buffers are the same ModelOpt NVFP4 format, sodecode_modelopt_nvfp4reuses them as-is).Type of change
Bug fix
Verification
tensorrt-edgellm-export Qwen3.6-35B-A3B-NVFP4 --skip-audio --skip-visualcompletes successfully after the fix (40-layer MoE repacked into
Nvfp4MoePlugin).Compatibility
Backward compatible
Checklist
pre-commit run --all-filesandfixed any reported issues.