Fix blockwise scale indexing under split-K (#3537) - #3615
Open
amacharla15 wants to merge 1 commit into
Open
amacharla15 wants to merge 1 commit into
amacharla15 wants to merge 1 commit into
Conversation
amacharla15
force-pushed
the
splitk-blockwise-repro
branch
3 times, most recently
from
September 12, 2026 03:43
ab5e948 to
6ee720d
Compare
amacharla15
force-pushed
the
splitk-blockwise-repro
branch
from
September 12, 2026 03:45
6ee720d to
092a003
Compare
Author
Contributor
|
@jackkosaian , could you please take a look? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3537
Under serial split-K, the blockwise scale lookup used the threadblock-local
K-tile counter (
k_iter_idx) with no contribution from the slice's position inthe global K range, so every slice applied slice-0's scale factors to its own
partial products.
Reproduction
M=N=256, K=512, scale block 128 (4 K-blocks), FP8 e4m3 with all-ones A and B,
kSplitKSerial = true, and powers-of-two scales per K-block:scale_A row 0: 1, 2, 4, 8 scale_B row 0: 1, 2, 4, 8
scale_A row 1: 2, 4, 8, 16 scale_B row 1: 2, 4, 8, 16
For output element (0,0) the correct value is
128 × (1·1 + 2·2 + 4·4 + 8·8) = 128 × 85 = 10880.The observed 1280 is exactly
2 × 128 × (1·1 + 2·2)— slice 1 re-appliedK-blocks 0 and 1 to data from blocks 2 and 3. All 65536 output elements were
affected.
can_implement,initializeand the kernel launch all returnedkSuccess, so the mis-scaled result was reported as success.Change
Thread the slice's K-tile base
(
threadblock_tile_offset.k() * params.gemm_k_size / Mma::Shape::kK) fromgemm_universal_blockwise.hdown throughoperator()→gemm_iters→mac_loop_iter, and add it tok_block_idxbefore the existing clamp. Theclamp is retained as a bounds guard. With
split_k_slices == 1the offset iszero, so existing behaviour is unchanged.
Coverage gap
The existing coverage on this path,
gemm_f8t_f8n_bf16t_tensor_op_f32_blockwise_sm89.cu,used K=128 with a single scale block and no split-K, so it could not observe the
mis-indexing. This PR extends that file to use K=512 with 4 scale blocks and
distinct per-block scales, and exercises
split_k_slicesof 1, 2 and 4 underboth threadblock swizzles.
Testing
RTX 4090 (SM89), CUDA 12.4.
cutlass_test_unit_gemm_device_tensorop_sm89_blockwisepasses withsplit_k_slicesof 1, 2 and 4.