Skip to content

cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sb32445 wants to merge 3 commits into
base: prism
Choose a base branch
from
Open

cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
431509d
cuda: use the MMA flash attention kernel for GQA above 4 with quantiz…
sb32445 Oct 2, 2026
069cd48
cuda: remove the GQA/MMA and vector-kernel query-limit switches
sb32445 Oct 3, 2026
d01570f
cuda : keep the vector FA kernel when MMA cannot read quantized K/V i…
sb32445 Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ggml/src/ggml-cuda/fattn.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,9 @@ static best_fattn_kernel ggml_cuda_get_best_fattn_kernel(const int device, const
}
} else {
if (cc >= GGML_CUDA_CC_ADA_LOVELACE) {
if (Q->ne[1] <= 2) {
// Above GQA 4 the MMA kernel is faster on Ada (same rule as for F16 K/V above), but only when it reads K/V in place.
// Otherwise it converts the whole cache to f16 on every call, so keep the vector kernel.
if (Q->ne[1] <= 2 && (gqa_ratio <= 4 || !ggml_cuda_fattn_mma_kv_native_supported(dst))) {
return BEST_FATTN_KERNEL_VEC;
}
} else {
Expand Down