torad-labs/llama.cpp

A llama.cpp fork with CUDA and server changes for RTX 50-series cards, on PrismML's llama.cpp and ggml-org/llama.cpp. On one 16 GB card, Ternary Bonsai 2 27B decodes at 35.5 -> 67.3 tok/s at 131K context against PrismML's release, and at 104.8 tok/s served with the draft head. It is for people running that model on one 16 GB RTX 50-series card.

Why use this fork

• Ternary Bonsai 2 27B is PrismML's ternary Qwen3.8-27B, PQ2_0 at 2.13 bits per weight, 7.66 GB with its draft head. It fits one 16 GB card, with a 262,144-token window.
• The gap is at long context. At 131,072 tokens, prefill goes from 547 to 1,104 tok/s and decode from 35.5 to 67.3, against PrismML's release.
• With the draft head, served decode is 104.8 tok/s. In the same session PrismML's release is at 75.0 tok/s. That release refuses this pack's head at load.
• Each change has an off switch and a measurement. The list is TORAD.md.
• Claude Code can use the server through splice.

Headline

One RTX 5070 Ti (16 GB), the public pack, PrismML prism-b10709-9a9394a against this fork at c008fe8. llama-bench, q4_0 K/V, flash attention on, no draft head, 3 repetitions.

context	prefill pp512, Prism -> fork	decode tg128, Prism -> fork
0	1,888 -> 2,164 tok/s	85.8 -> 85.8 tok/s
16,384	1,626 -> 1,994	71.2 -> 81.9
65,536	874 -> 1,479	50.3 -> 75.0
131,072	547 -> 1,104	35.5 -> 67.3

How this was measured

Both binaries were built with the flags in Build. The fork side is the published engine-c008fe8 build. The host had been quiet for two minutes (1-minute load under 6; at most 4.4 during the run, quiet-rerun.sh). bench-matrix.sh: llama-bench, q4_0 K/V, flash attention on, no draft head on either side, 3 repetitions.

Depth-0 decode in this table is 85.8 tok/s on both sides. A 5-repetition A-B-A-B on the Bonsai page gives Prism 88.4 / 88.2 and the fork 87.1 / 87.1 tok/s. With every off switch set, the fork does 87.4 tok/s, so the gap sits outside those switches.

Served, 2026-09-22, fork at c54ace8. One conversation, a 65,536-token window, six prompts of 512 tokens, greedy.

server	decode	against Prism, same session
fork, MTP draft head	104.8 tok/s (85.8 on prose, 120.8 on step-by-step reasoning)	1.40x (Prism 75.0)
fork, no draft head, recurrent state in f32, as Prism keeps it	77.0 tok/s	-1.5%, within the drift (Prism 78.2)
fork, no draft head, recurrent state in q8_0 (-cts q8_0, 3.4x less state VRAM)	71.3 tok/s	-4.9% (Prism 75.0)

How this was measured

Two runs each, with Prism measured in the same session. The 2026-09-24 served rows ran at host load 20-50. That load moves decode by 8-12 tok/s, so the table uses the earlier run.

Prism b10709 refuses this pack's draft head at load, so it has no MTP row. On the fork, 47% of drafted tokens were accepted on prose and 85% on arithmetic reasoning. The command for that row is in Benchmark. Per-prompt rows, logs, and the rest of the method are on the Bonsai page.

Quick start

CUDA 12.8 or newer. The build targets the GPU in the machine.

1. Build

git clone https://github.com/torad-labs/llama.cpp && cd llama.cpp
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON -DGGML_NATIVE=ON \
  -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=OFF -DGGML_CUDA_GRAPHS=ON \
  -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TESTS=OFF
cmake --build build --target llama-server llama-bench -j

With CMake 3.24 or newer, -DGGML_NATIVE=ON sets CMAKE_CUDA_ARCHITECTURES to native when that value is still unset: the GPUs in the build machine. To build on one machine for another card, set CMAKE_CUDA_ARCHITECTURES to that card's architecture: 120a for RTX 50-series and RTX PRO Blackwell, 121a for DGX Spark GB10, 100a for B200, 90a for H100, 89 for RTX 40-series, 86 for RTX 30-series. The PQ2_0 tensor-core mat-vec launches only on sm_120, and only when the binary includes an sm_120 image. Use 120a for that card. The CUDA build rewrites a plain 120 to 120a.

2. Fetch the model

Apache-2.0. The file is ProCreations' Ternary-Bonsai-2-27B-MTP: PrismML's 851 tensors byte for byte, plus a 15-tensor MTP draft head in Q8_0.

hf download ProCreations/Ternary-Bonsai-2-27B-MTP Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf \
  --revision efffdea64c1f9e93cc7fa6bb24f72ae9d66ecf51 --local-dir models
echo "3cb3f0056d2e34ee44245a64396004a21f8492573d6ce1266ec4b7222c131dd4  models/Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf" | sha256sum -c

3. Serve

For a 16 GB card, four conversations share one 294,912-token pool. Each may use the model's full 262,144-token window.

build/bin/llama-server -m models/Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf --alias bonsai-2-27b \
  -ngl 99 -fa on -c 294912 -np 4 --kv-unified --jinja \
  --cache-type-k q4_0 --cache-type-v q4_0 -cts q8_0 --attn-mask-bits \
  --chat-template-file docs/torad/bonsai-27b/chat-template.jinja --reasoning-format deepseek \
  --spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0 \
  --checkpoint-every 16384 --cache-ram 8192 \
  --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

--attn-mask-bits is CUDA and CPU only. Hardware support has the detail.

chat-template.jinja is the model's own template with one change: a system message after the first turn renders as a system turn. Claude Code sends those.

On a bigger card, keep each conversation's window at 262,144 and add conversations to the shared pool.

To put the server behind Claude Code, use splice. The Bonsai page has the splice.toml and the three settings that were measured.

4. Benchmark

The depth table, on the binary from step 1:

docs/torad/bonsai-27b/bench-matrix.sh build/bin models/Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf

That runs llama-bench -ngl 99 -fa 1 -ctk q4_0 -ctv q4_0 -p 512 -n 128 at depths 0, 16384, 65536, and 131072, three repetitions, no draft head.

The 104.8 tok/s row is bench-server.sh against the fork-mtp leg of served.sh. Flags that row depends on: -ngl 99, -fa on, --cache-type-k q4_0, --cache-type-v q4_0, -cts q8_0, --attn-mask-bits, --spec-type draft-mtp, --spec-draft-n-max 2, -ctkd q4_0, -ctvd q4_0, and -c 65536 -np 1. The client sends temperature 0 and top_k 1, so the row is greedy. The recorded script used port 18090. Start the server, then run the client in another shell:

build/bin/llama-server -m models/Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf --alias bonsai-2-27b \
  -ngl 99 -fa on -c 65536 -np 1 --jinja \
  --cache-type-k q4_0 --cache-type-v q4_0 -cts q8_0 --attn-mask-bits \
  --chat-template-file docs/torad/bonsai-27b/chat-template.jinja --reasoning-format deepseek \
  --spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0 \
  --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

docs/torad/bonsai-27b/bench-server.sh 8080 fork-mtp

Step 3 is the command to leave running on a 16 GB card. It uses -c 294912 -np 4 and also passes --kv-unified, --checkpoint-every 16384, and --cache-ram 8192, which keep several conversations resident. The server command above is the one the 104.8 tok/s row measured.

Both commands leave --kv-mean-center unset, and so did the headline runs. When to set it is the next section.

q4_0 cache

The message on 60feea0 says the native q4_0 flash-attention read needs --kv-mean-center for a usable 4-bit K cache at long context. The kernel selects that read in ggml_cuda_fattn_mma_raw from the cache type (q4_0 K and V), head size 256, and GQA above 4. GGML_CUDA_FATTN_Q4_0_LEGACY=1 is the off switch.

--kv-mean-center FNAME is a separate option. The argument is a bias file from tools/kv-mean-center. After the file is loaded, cpy_k() subtracts that bias before the q4_0 store. Context creation accepts the flag when --cache-type-k is q4_0. Set it when you have a bias calibrated with that same cache type (docs/kv-mean-center.md).

A quantized K cache whose head size is a multiple of 64 is Hadamard-rotated unless LLAMA_ATTN_ROT_DISABLE=1. This model's head size is 256, and the scripts in docs/torad/bonsai-27b/runs/ leave that variable unset, so the headline runs rotated K. On a hybrid-attention model, the tool's README reports mean KLD against an f16 K cache of 0.00144 with rotation alone and 0.00111 with rotation plus a bias calibrated with -ctk q4_0. A bias recorded in the other basis is rejected at load.

Hardware support

All published numbers come from RTX 50-series and RTX PRO 6000 cards, sm_120 (GeForce and RTX PRO Blackwell), where the defaults were tuned.

optimization	runs on	elsewhere
int8-query tensor-core flash attention for a q4_0 K/V cache	NVIDIA sm_80 and newer (Ampere, Ada, Hopper, B200, RTX 50-series, GB10), head size 256, GQA above 4	upstream flash-attention kernels
chunked Gated DeltaNet prefill	NVIDIA sm_80 and newer, head size 128, prompts of 128 or more tokens	upstream recurrent kernel
MTP draft-head Hadamard fix, --cache-type-s, small-float mat-vec instead of cuBLAS	The Hadamard fix and --cache-type-s run on every backend. The small-float mat-vec runs on every CUDA GPU.	The Hadamard fix and --cache-type-s use that same path on every backend. On CUDA, a larger float product stays on cuBLAS. Other backends use their own matmul for that product.
paced L2 prefetch and launch overlap (PDL)	NVIDIA sm_90 and newer (Hopper, B200, RTX 50-series, GB10)	Kernels run in program order. Each kernel fetches its own weights.
PQ2_0 tensor-core mat-vec with a TMA pipeline, evict_first, and L2 prefetch	sm_120 (RTX 5070 Ti, 5080, 5090, RTX PRO Blackwell)	The CUDA-core PQ2_0 kernel on other NVIDIA GPUs, including B200 (sm_100) and DGX Spark GB10 (sm_121). That kernel works and is slower. PQ2_0 also runs on CPU and Metal. Vulkan and SYCL have no PQ2_0 kernel.
--attn-mask-bits	CUDA and CPU. Off by default.	Vulkan refuses the bit-packed mask and runs that attention op on the CPU. Metal does not check the mask type, so leave the flag off there.

What's different from upstream

The changes are general CUDA and server changes. Each row is one commit. The Bonsai page has the measurement on this model.

change	commit	off switch
Tensor-core flash attention reads a q4_0 K/V cache directly, with int8 QK	60feea0	GGML_CUDA_FATTN_Q4_0_LEGACY=1
Chunked tensor-core Gated DeltaNet prefill, a port of ggml-org#26001	b120f63	GGML_CUDA_GDN_CHUNKED=0
The MTP draft head reads the pack's Hadamard-latent token table through the inverse rotation	514c53c	omit --spec-type draft-mtp
Recurrent state cache in a narrower type, -cts q8_0. The update math stays f32	a941add	omit -cts (f32)
Attention mask as one bit per KV cell, --attn-mask-bits	9e358e6	omit the flag
A small float matrix the tiled kernel cannot take goes to the vector kernel	55b7db6	GGML_CUDA_MMVF_UNTILED_LEGACY=1
GeForce Blackwell takes the transposed conv-state concat and the float4 scale	d371dff	GGML_CUDA_CONCAT_TRANSPOSE_SM120_LEGACY=1, GGML_CUDA_SCALE_VEC4_SM120_LEGACY=1
A pinned context checkpoint every N prompt tokens, and one at the token a prompt forked, --checkpoint-every	aa8ef36	omit the flag

aa8ef36 is the parent of 60feea0. The Bonsai page lists the checkpoint with 60feea0, which is where both landed on this branch.

The full list is TORAD.md.

Contributing

Issues and pull requests for this fork: issues, pull requests.

A change here is one commit with its own measurement and its own off switch, so it can be rebased, dropped, or sent upstream on its own. TORAD.md is that list. CONTRIBUTING.md is the upstream llama.cpp rule set, including the note on disclosing AI-written changes.

License

MIT. Copyright (c) 2023-2026 The ggml authors. Same license as ggml-org/llama.cpp. Third-party notices are in the upstream README.

Upstream

PrismML-Eng/llama.cpp is the base. This fork merges that line when it takes a newer base (prism remote). PrismML tracks ggml-org/llama.cpp.

A fix that belongs upstream can go as its own commit. Opening it there is a separate pull request. The maintained description is the last section of TORAD.md.

Documentation

• Ternary Bonsai 2 27B: the runs, the method, splice
• Build
• llama-server
• Docker
• Upstream README: what llama.cpp is, backends, tools, and PrismML's note on which GGUF to load

Made by Torad

Torad