cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth

[professorpalmer]

Summary

Bonsai 2 27B at its full 262,144-token window with q8_0 K/V on a 12 GB RTX 4070, with MTP drafting at every depth. #221 has merged (5244cea); this branch is now its own 6 commits on current prism (445fa82), replayed without conflicts (git range-diff identical to the stacked version).

q8_0 at 262k is 9.1 GB of KV and does not fit next to the weights; #221 got there with q4_0 (mean KLD 0.00218 vs f16 K/V, 97.9% top-1 agreement, against 0.00017 / 99.4% for q8_0 on this model).

RTX 4070 (sm_89, PCIe 4.0 x16, GDDR6X +1500 MHz as in #221's table), llama-server, one slot, greedy, MTP head, genuine prefill:

depth	prefill tok/s	decode tok/s	#221 recipe at that depth
4k	1,100	83
16k	1,100	106
32k	918	100	47.6 (96k/q8_0, drafting cut off at 24k)
64k	724	87	36.9
112k (last VRAM position)	539	70	q8_0 does not fit
131k	374	41	q8_0 does not fit
180k	298	27	q8_0 does not fit
258k	229	14.7	q8_0 does not fit

Commits

1. llama/cuda: tiered KV cache (--kv-vram-cells). llama_context_params.n_kv_vram_cells / --kv-vram-cells N keeps cells [0, N) of each layer's K/V in device memory and the rest in pinned host memory mapped into the same device range with CUDA VMM (cuMemCreate with CU_MEM_LOCATION_TYPE_HOST); ggml_backend_cuda_tier_buffer_type is exported through the registry. Kernels unchanged; output bit-identical to an all-device cache. When an attention op reaches the host tail, the used rows are copied into a VRAM staging buffer by the copy engine (a second VMM alias maps the same VRAM head plus the staging buffer) and the op reads the alias: prefill reads the tail once per op instead of once per query tile (146 -> 280 tok/s at 180k), decode moves it by DMA (+28% at 180k). Plumbed through llama_cparams into the standard KV cache and the hybrid memory with a trailing defaulted constructor parameter; other memory types untouched. Works on Windows/WDDM.
2. cuda: quantized-KV GQA decode on the in-place MMA flash-attention kernel. Up to 8 queries with q4_0/q8_0 K/V and GQA > 4 take the MMA kernel from e5cd0f3 instead of the vector kernel, which fetches each K/V row once per Q head (6x here). 4k 78.1 -> 86.9, 16k 77.7 -> 111.6 tok/s. GGML_CUDA_FA_MMA_DECODE_MIN_KV (default 256, 0 = off).
3. cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns. The vector path's KV split no longer depends on the occupancy of the instance that runs; 5-8 column PTQ1_0 batches stay on the PT mat-vec (a 5-column verify on MMQ did not match single decode). GGML_CUDA_PTQ1_MMVQ_MAX overrides the crossover; the default of 4 is confirmed (pp5 172 MMQ vs 162 mat-vec).
4. speculative: --spec-draft-window and --spec-draft-n-max-tail. The draft (MTP) context keeps only the last N rows (the server drops older ones before each batch; the context is sized for the window, so cells are reused and a draft pass costs the same at any depth). With (2) drafting pays at every depth, which retires the --spec-draft-depth-max 24576 cutoff from cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221 (the flag stays, default 0): 32k 54.5 -> 103.6, 64k 48.0 -> 90.1 tok/s, acceptance unchanged. --spec-draft-n-max-tail sets the draft size from --kv-vram-cells on (4 vs 2: +26% code / +15% prose at 180k); the MTP draft loop now honours the per-draft cap.
5. server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens-floor. Bonsai 2's template raises on reasoning_effort: "high" (Cline, Kilo, Open WebUI: HTTP 500 on every request); an effort word outside the allow list becomes the fallback. With thinking on, a client max_tokens below the floor is raised to it (256-4096-token app caps end a thinking model mid-thought). Both off by default.
6. cuda: GGML_CUDA_OP_TIMING=1 per-node GPU time breakdown with graphs off. Diagnostics.

Correctness

• test-backend-ops -b CUDA0: FLASH_ATTN_EXT 2994/2994 and MUL_MAT 1516/1516, default and with GGML_CUDA_BATCH_INVARIANT=1, on the stacked head c8b8993 and again on the rebased head (GATED_DELTA_NET 39/39, GET_ROWS 119/119 too). MUL_MAT_ID is 1034/1035 on both: the intermittent m=70, n=1 case bri-prism reported on cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221, a store-guard bug already in prism, fixed in cuda: bound the one-column MUL_MAT_ID mat-vec store by the expert row count #295.
• Tiered vs all-VRAM KV, same flags, drafting on, CLI flags: identical sha256 of 256-token code and prose continuations at 4k, 20k and 40k. Rechecked on the rebased head: --kv-vram-cells 2048 vs all-VRAM, q8_0, drafting on, greedy, 256-token code and prose continuations at 4096 / 20480 / 40960 tokens: 6/6 identical.
• On cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221's identity claim: greedy draft-on is not always byte-identical to draft-off, with the stock kernels too. The FA KV split follows the kernel instance (1-query vs 3-5 query tiles on the MMA path) and the padded KV length (a verify batch can move it one 256-tile ahead of a later single decode). Measured: equal at 4k/20k and different at 40k with the vector path below 32k; different at 4k and 40k with (2). Rounding-level; every continuation is a valid greedy decode. (3) keeps the weight path exact. A single long-prefix match (bri-prism's 47,859-token check on cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221) is consistent with this: the split only moves when a verify batch crosses a 256-tile boundary ahead of a later single decode, so whether a given prompt diverges depends on where it lands.

End-to-end quality (HumanEval 164, tests executed, server's own sampling)

Killy's HumanEval plates replayed against this branch's server (262k window, q8_0 KV tiered, MTP): medium 161/164 (his best cell on PQ2_0: 160-161), thinking off 138. Harness-proofing on the same server: reasoning_effort: "high" with a 256-token cap 160/164, against 0/20 (HTTP 500) with it off; a 4096-token cap 157/164, against 148/164 with it off. Samples and summaries: artifacts/eval/killy_20260927.

Killy's voxel-pagoda plate (037P) at medium, 6 samples on this server: 3 correct on the first try, 6 of 6 after a render-check-repair loop (bench/pagoda_plate.py --repair: run the page headless, send back console errors / floating voxels / out-of-view, ask for a fix; 1-2 rounds). The first-try misses were one-token slips in the model's JavaScript, not runtime faults. Details.

All charts in one image: mega.png (decode and prefill by depth, KV precision, iGPU, identity matrix, HumanEval replays, pagoda).

Measured and not adopted

idea	result
K mean-centering on q4_0 / q8_0 with the default Hadamard KV rotation	q4_0 KLD 0.00206 vs 0.00218, max KLD 2.87 vs 0.94; q8_0 unchanged
GGML_CUDA_GRAPH_OPT=1	tg128 66.1 vs 67.9
one MMA tile for all <= 8-query batches (invariance)	-7% at 64k, and the KV-length dependence remains
draft cache fully in VRAM, paid for with a lower KV line	16.0 vs 17.2 tok/s at 180k

Notes

• On WDDM the VRAM line must stay below the point where Windows demotes a background process: ~1.3 GB below free VRAM with the display on the card, ~1.0 GB with the display on an iGPU (measured by 10-minute soaks). Past the line decode is PCIe-bound.
• Serving recipe, all receipts, identity matrix and HumanEval replays: https://github.com/professorpalmer/bonsai-ada-surgery/blob/main/docs/Q8_FULL_CONTEXT.md

🤖 Generated with Claude Code

[professorpalmer]

Rebased now that #221 has merged (5244cea). The branch is just its own 6 commits on prism 445fa82, replayed without conflicts; git range-diff shows every commit identical to the stacked version (c8b8993).

This head built clean on Windows (sm_89, CUDA 13, VS 2022): llama-server, test-backend-ops, test-mtp-catchup-batch (prints ok). I have not rerun test-backend-ops on the GPU for this head yet because the card is serving; the FA/MUL_MAT counts in the description are from c8b8993 and are labelled that way.

[professorpalmer]

Reran the checks on the rebased head (3164035) now that the card is free:

• test-backend-ops -b CUDA0, default and GGML_CUDA_BATCH_INVARIANT=1: FLASH_ATTN_EXT 2994/2994, MUL_MAT 1516/1516, GATED_DELTA_NET 39/39, GET_ROWS 119/119. MUL_MAT_ID 1034/1035: the one failure is the intermittent m=70, n=1 case from cuda: Bonsai 2 27B at the full 262k window with the MTP head on 12 GB Ada: #218 + #215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) #221, a store-guard bug that is already in prism and not touched here. Fix in cuda: bound the one-column MUL_MAT_ID mat-vec store by the expert row count #295.
• Tiered vs all-VRAM KV (--kv-vram-cells 2048 so every depth crosses into the host tail), q8_0, MTP drafting on, greedy: 256-token code and prose continuations at 4096 / 20480 / 40960 tokens, 6/6 byte-identical.
• test-mtp-catchup-batch: ok.

Description updated with these.

[professorpalmer]

Split into one PR per feature so each can land on its own, as asked on #317: #319 (tiered KV cache), #320 (MTP drafting window and tail), #321 (server reasoning flags), #322 (batch-invariant FA split and PTQ1_0 mat-vec crossover), #323 (op timing, diagnostics). The quantized-KV GQA-decode FA commit is not re-submitted; its ggml_cuda_fattn_mma_kv_native_supported guard goes into #307. The description above stays as the combined write-up and receipts.