Repository navigation
INT8 attention: keep large values finite (0.17.1) - #28
Merged
Merged
Conversation
… (0.17.1) A 64-key FP16 tile of P.V reaches 64 * max|v|. Vimoo 1.0 Lite reaches |v| = 7136 in block 38 late in sampling, so the tile overflowed to inf and the video turned to NaN. Heads above |v| = 512 are now divided by a power of two before the FP16 conversion and the output is multiplied back; the scale is exact, so heads within the limit give the same bits as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INT8 attention (
int8_attention_with_triton) accumulates each 64-key tile of P·V in FP16. With equal scores a tile sum reaches 64 × max|v|, which overflows FP16 when |v| exceeds about 1023. Vimoo 1.0 Lite reaches |v| = 7136 (block 38, about step 29 of 50, 1280×720, 61 frames): the output became inf/NaN and the whole video was black.Change
V_LIMIT = 512are divided by a power of two before the FP16 conversion, and the kernel multiplies the output by the same factor.Checks
tests/test_fused_dit_kernels.py::test_int8_attention_keeps_large_values_finite: fails on 0.17.0 (non-finite output) and passes with this change.tests/test_fused_dit_kernels.py,tests/test_fused_family_kernels.py,tests/test_fused_vimoo.pyon an RTX PRO 6000 (Torch 2.12.1, CUDA 13.0): pass.ruff check .,pytest: pass locally.