cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079)

[cheese-cakee]

Overview

Fixes #267.

Backport of upstream ggml-org#28079 (5a4d0fe, by @pwilkin). With the default CUDA build, a K/V type pair without a compiled vector kernel (for example q8_0/q4_0) is now converted to f16 on the GPU with a one-time warning, instead of being rejected by the CUDA backend and running flash attention on the CPU without any message:

ggml_cuda_flash_attn_ext_vec: no FlashAttention vector kernel compiled for K/V types q8_0-q4_0, converting K and V to f16 instead (slow). Add "q8_0-q4_0" to GGML_CUDA_FA_QUANTS to compile it.

GGML_CUDA_FA_QUANTS selects which pairs get native kernels (default q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16, or all). GGML_CUDA_FA_ALL_QUANTS stays as a deprecated alias for all.

Additional information

The cherry-pick conflicted only in the docs/build.md options table (kept GGML_CUDA_PEER_MAX_BATCH_SIZE). fattn.cu merged without conflicts. The in-place q4_0/q8_0 MMA path from #221 still requires K->type == V->type, so mixed pairs take the f16 conversion path.

Testing on RTX 4050 Laptop (cc 8.9), CUDA 12.6, Linux (WSL2), -DCMAKE_CUDA_ARCHITECTURES=89:

• test-backend-ops -o FLASH_ATTN_EXT: prism default build 2994/2994 with all mixed cases "not supported"; this PR default build 4007/4007, mixed cases pass against CPU. prism with GGML_CUDA_FA_ALL_QUANTS=ON is also 4007/4007.
• Qwen3-0.6B-Q8_0, -ngl 99 -fa 1, GGML_SCHED_DEBUG=2: on prism the FLASH_ATTN_EXT nodes of all 28 layers run on CPU for q8_0/q4_0, f16/q8_0 and q4_0/q8_0; with this PR they run on CUDA0, and the warning is printed once per process.
• The shape from cuda: avoid short-query MMA FA crash with quantized V #189 (hs 256, GQA 6, kv 1280, 2-5 queries, f16 K with q8_0/q5_1/q4_0 V) passes.
• cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307, cuda: smaller KV tile for the 8-column MMA flash attention config (head size 256) #308 and cuda: avoid short-query MMA FA crash with quantized V #189 apply cleanly on top of this PR.
• Not tested: HIP and MUSA (their CMake files change the same way), other GPU architectures.

llama-bench, Qwen3-0.6B-Q8_0, -ngl 99 -fa 1 -t 8 -r 3, q8_0 K / q4_0 V, t/s, median of 5 alternating rounds:

test	prism default (FA on CPU)	this PR, default (f16 conversion)	prism + GGML_CUDA_FA_ALL_QUANTS=ON
pp512	888	13937	14378
pp512 @ d4096	47	8631	8881
tg128	78.7	190.7	186.5
tg128 @ d4096	17.3	99.6	143.3

The f16 conversion is much faster than the CPU fallback, but decode at long context is still slower than a compiled pair, which is what the warning says. For q8_0/q8_0 (already supported) the three builds are within run-to-run spread, except pp512 at depth 0, which varies by up to 15% between runs on this laptop.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI was used for the backport and for running the tests; I checked and verified the change and the results afterwards.

[bri-prism]

@cheese-cakee @sb32445 @professorpalmer @Wyzix33, several open PRs touch flash attention with quantized K/V: #317 (upstream backport), #307 (MMA for GQA above 4), the FA commit inside #285, and #189. Our plan is to land #317 first, since it matches upstream.

• @Wyzix33, cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 handles the short-query shape from cuda: avoid short-query MMA FA crash with quantized V #189. In testing cuda: avoid short-query MMA FA crash with quantized V #189 also hit an illegal memory access in FLASH_ATTN_EXT and lowered MTP decode, so we'd like to close it in favor of cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 unless you see a case cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 misses.
• @sb32445 and @professorpalmer, cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 and the FA commit in cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285 change the same routing. cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285's version already checks ggml_cuda_fattn_mma_kv_native_supported(dst), which cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 needs too: without it, types the MMA kernel can't handle in place (head size 64, q5_x, mixed K/V after cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317) would convert the whole cache to f16 on every decode step. Could you agree on one change, probably cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 with that guard?
• @professorpalmer, cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285 bundles several independent features. Splitting it (tiered KV, MTP depth, server flags) would let the pieces land separately.

[professorpalmer]

Agreed on the order: #317 first.

On #307 vs the FA commit in #285 (quantized-KV GQA decode on the in-place MMA kernel): one change, #307 with the ggml_cuda_fattn_mma_kv_native_supported(dst) guard, is fine by me, and I am not opening a separate PR for that commit. @sb32445, if you would rather take the guard as a patch against your branch than fold it in yourself, say so and I will send it.

On splitting #285: the branch was six commits, each of which applies cleanly on its own and on the current prism head (eaecb50c7), so #285 is now replaced by one PR per feature, each with the receipts that belong to it:

• llama/cuda: tiered KV cache (--kv-vram-cells): VRAM head, pinned-host tail in one VMM range #319 llama/cuda: tiered KV cache (--kv-vram-cells): VRAM head, pinned-host tail in one VMM range
• speculative: --spec-draft-window and --spec-draft-n-max-tail (MTP drafting at every depth) #320 speculative: --spec-draft-window and --spec-draft-n-max-tail: MTP drafting at every depth
• server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens-floor #321 server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens-floor
• cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns #322 cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns
• cuda: GGML_CUDA_OP_TIMING=1 per-node GPU time breakdown #323 cuda: GGML_CUDA_OP_TIMING=1 per-node GPU time breakdown (diagnostics; drop it if you would rather not carry it)

The FA GQA-decode commit stays out, per the above. On the two speculative-deferral defects from the #221 review: both are fixed in prism as merged (begin() clears catchup_failed with the other per-task state; draft() marks the stash failed on a failed decode instead of dropping it), so none of these PRs touch that path.