Does llama.cpp Support TurboQuant? Not in Mainline — How to Run turbo4 in a Fork

Does llama.cpp Support TurboQuant? Not in Mainline — How to Run turbo4 in a Fork#

Short answer: not in mainline llama.cpp. The upstream pull request that added TurboQuant cache types (#21089, CPU-only tbq3_0/tbq4_0) was closed without merging on 2 June 2026, so official builds have no turbo cache types. To use TurboQuant today you have three options:

• A llama.cpp fork: TheTom/llama-cpp-turboquant (turbo2 to turbo6, with turbo code for CUDA, Metal, Vulkan and SYCL) or spiritbuun/buun-llama-cpp (CUDA and ROCm, formerly llama-cpp-turboquant-cuda).
• vLLM v0.20 or later: --kv-cache-dtype turboquant_4bit_nc and three other presets, the only production engine that ships it. vLLM's own study still recommends FP8 unless you need the extra capacity (Part 3).
• HuggingFace transformers: the independent turboquant pip package. It is useful for studying the error, but its 0.2.0 release held more cache memory than the default cache in our test (section 3).

In a fork, pick turbo4, not turbo3. On our A100, turbo4 matched q4_0 quality (+0.65% perplexity vs f16) and turbo3 tripled perplexity at 16K and 32K context (Part 5).

Correction (October 2026): earlier versions of this page recommended turbo3, said values were more sensitive than keys (our measurements show the opposite), recommended 3-bit caches for 13B+ models without evidence, showed an 8B benchmark table and MSE outputs that we had not measured, gave 70B cache sizes that were far too small (27.3 GB for f16 at 128K; the real figure is 40 GiB), which made a 70B at 128K look like it fit on a 48 GB card, and its script used 64 KV heads instead of 8, used a llama-bench command that does not compare types, presented the turboquant pip cache as a memory saver, and used a residual_length argument that the package does not have. An August note already corrected the claim that mainline llama.cpp had merged TurboQuant. All of these are fixed below; the numbers now come from our own runs, the fork READMEs, or the posts they link to.

If you came here for mainline llama.cpp's own KV cache quantization (-ctk q8_0, q4_0, -fa on) rather than TurboQuant, the measured answer is in llama.cpp KV Cache Quantization, Measured on One A100, and the flags to copy are in the KV cache quantization guide.

1. Which route to take#

Ollama is in the first row for the same reason as mainline: its TurboQuant pull request (#15505) was closed in May 2026. The full status of each project is in Part 3: TurboQuant Status Check.

2. TurboQuant in a llama.cpp fork#

The commands below follow each fork's own docs. Our measurements in this section come from the spiritbuun fork (now buun) as of August 2026; we have not built TheTom's fork ourselves.

Building#

TheTom's fork keeps the TurboQuant work on its default branch, feature/turboquant-kv-cache.

git clone https://github.com/TheTom/llama-cpp-turboquant.git
cd llama-cpp-turboquant
cmake -B build -DGGML_CUDA=ON     # NVIDIA
# cmake -B build -DGGML_METAL=ON  # Apple Silicon
cmake --build build --config Release -j

The executables land in build/bin/.

A GGUF model#

Any GGUF works; TurboQuant changes the cache, not the weights. For example, Llama 3.1 70B Instruct (128K context) at Q4_K_M (39.6 GiB):

hf download bartowski/Meta-Llama-3.1-70B-Instruct-GGUF \
  Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf --local-dir ./models

hf is the current name of the HuggingFace CLI; older guides call it huggingface-cli.

Running the server#

./build/bin/llama-server -m ./models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf \
  -ctk turbo4 -ctv turbo4 \
  -c 131072 -fa on -ngl 99 --port 8080

The server speaks the OpenAI API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 256}'

The buun fork's default cache moves#

buun-llama-cpp turns on a variable bit-rate cache (VBR) by default. With no cache flags the cache starts at f16, but once memory runs short it quantizes layers one at a time down a ladder (f16 → turbo8 → turbo4 → …), stopping at a turbo4 floor. So a run without flags is not a fixed f16 baseline: how much of it is compressed depends on context and free VRAM. Pass -ctk f16 -ctv f16 for a real f16 baseline, or -ctk turbo4 -ctv turbo4 to pin one type.

Comparing on your own GPU#

llama-bench builds a cross product from comma lists, so -ctk f16,turbo4 -ctv f16,turbo4 also runs the mixed pairs. To compare matched pairs only, loop:

for t in f16 q8_0 turbo4 turbo3; do
  ./build/bin/llama-bench -m model.gguf -fa 1 -ngl 99 \
    -ctk $t -ctv $t -p 512 -n 128 -d 0,8192,32768,65536
done

-d measures decode with that many tokens already in the cache, which is where KV bandwidth shows. Our A100 numbers come from the spiritbuun fork (now buun) as of August 2026, not from TheTom's fork: at 64K depth on Qwen3-8B, f16 79.6 tokens/s, turbo4 57.8 (73% of f16), q8_0 22.8 (29%). The full table is in Part 5.

3. TurboQuant in HuggingFace transformers#

The turboquant package on PyPI (from back2matching/turboquant, independent of Google) gives you a quantizer and a cache class. We ran equivalents of both examples below on CPU on 2026-10-25 with turboquant 0.2.0, transformers 5.20.2 and torch 2.14.1.

pip install transformers torch accelerate "turboquant==0.2.0" "numpy<2.4"

Pin numpy<2.4. turboquant 0.2.0 calls np.trapz, which numpy 2.4 removed, so building a quantizer fails without the pin.

Quantize vectors and look at the error#

import torch
from turboquant import TurboQuantMSE

torch.manual_seed(0)
vectors = torch.randn(1024, 128)   # stand-in for K or V rows

for bits in [2, 3, 4]:
    tq = TurboQuantMSE(dim=128, bits=bits, device="cpu")
    indices, norms = tq.quantize(vectors)
    vectors_hat = tq.dequantize(indices, norms)
    mse = ((vectors - vectors_hat) ** 2).mean()
    rel = ((vectors - vectors_hat) ** 2).sum() / (vectors ** 2).sum()
    print(f"{bits}-bit MSE: {mse:.6f}  relative: {rel:.4f}")

Our output:

2-bit MSE: 0.129699  relative: 0.1293
3-bit MSE: 0.049991  relative: 0.0499
4-bit MSE: 0.019725  relative: 0.0197

Each extra bit cuts the error by roughly 2.5 times. A 5% relative error per vector at 3 bits looks small, but it is not a quality guarantee: attention ranks thousands of keys at once, and in llama.cpp a different 3-bit implementation, the fork's turbo3, tripled perplexity at 16K (section 5).

The cache class does not save memory in 0.2.0#

The package also offers a drop-in cache:

from turboquant import TurboQuantCache
outputs = model.generate(**inputs, past_key_values=TurboQuantCache(bits=4), use_cache=True)

It runs, but read the source before you rely on it for memory. In 0.2.0 each layer keeps the newest 128 tokens in full precision (fixed; there is no argument to change it), stores older tokens as one uint8 index per value plus a norm, and then, on every step, dequantizes all of them back into a full-precision copy that it keeps. We added up the tensors each cache object held after a 2,401-token prompt on Llama 3.2 1B Instruct (bf16):

The default cache matches the formula in section 4 (16 layers × 8 KV heads × 64 × 2 × 2 bytes × 2,404 tokens: the 2,401-token prompt plus three generated tokens). The TurboQuant cache held 1.56 times as much, because the dequantized copy sits next to the compressed one. Its own report counts only the compressed part, and bits=3 and bits=4 come out the same size because both store one byte per index. Two more things we checked:

• A run whose prompt plus generated tokens stays under 128 tokens is never quantized, so a short "The capital of France is" test with a few new tokens says nothing about quality. (Our first test was exactly that: 20 new tokens, and all three caches gave the same answer.)
• TurboQuantCache(bits=3, protected_layers=[0, -1]) and TurboQuantCache(bits=3, residual_length=256) both raise TypeError in 0.2.0. The GitHub main branch adds protected_layers but still builds the same dequantized copy; there is no release after it.

Use the package to see how TurboQuant's error behaves. To save memory, use a llama.cpp fork or vLLM.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

4. How much memory a fork saves#

Per token, the cache holds layers × KV heads × head dimension × 2 (K and V) × bytes per value. Use the KV head count (num_key_value_heads in config.json), not the attention head count. Llama 3.1 70B has 64 attention heads but 8 KV heads.

Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128) at 131,072 tokens:

The bits per value include the per-block scale. The turbo figures are TheTom's formats; buun's current turbo4 is 4.125 bits and its scalar turbo3 3.5 (README), so the two forks' caches differ slightly in size. In Part 5 the measured VRAM differences tracked these figures: at 32K on Qwen3-8B, turbo4 and q4_0 came out the same size (6.57 and 6.55 GB peak).

def kv_cache_gib(layers, kv_heads, head_dim, tokens, bits_per_value):
    return 2 * layers * kv_heads * head_dim * tokens * bits_per_value / 8 / 2**30

for name, bits in [("f16", 16), ("q8_0", 8.5), ("q4_0", 4.5), ("turbo4", 4.25), ("turbo3", 3.25)]:
    print(f"{name:7s} {kv_cache_gib(80, 8, 128, 131072, bits):5.1f} GiB")

Add the weights and a compute buffer to get the total: Q4_K_M weights (39.6 GiB) plus an f16 cache at 128K is about 80 GiB before buffers, which does not fit an 80 GB A100. With turbo4 it is about 50 GiB.

5. Which type to pick#

These recommendations come from Part 5: Qwen3-8B Q4_K_M on one A100, perplexity on wikitext-2.

The damage is on the key side: with keys at q8_0, 3-bit values stayed flat across context lengths. One model, one GPU and one fork build from August 2026, so treat the exact percentages as ours; the pattern (4-bit holds, 3-bit keys break with length) is what to check on your model and fork.

Measure perplexity at the context you serve. At 4K, turbo3 looked acceptable. At 16K it was not.

./build/bin/llama-perplexity -m model.gguf \
  -f wikitext-2-raw/wiki.test.raw \
  -ctk turbo4 -ctv turbo4 -fa on -c 16384 --chunks 3

scripts/get-wikitext-2.sh in the fork's source tree downloads the test file. Run the same command with -ctk f16 -ctv f16 and compare the two numbers on the same text.

6. Gotchas#

Mainline flags look the same#

-ctk turbo4 on an official build stops with Unsupported cache type. If it does, you are running a mainline binary, not the fork; check which llama-server is first on your PATH.

Flash attention#

The turbo kernels are written for the flash-attention path. Keep -fa on. The buun README says its default VBR cache requires flash attention and turns it on itself.

Metal#

On Apple Silicon, check that the cache is really on the GPU: run the server with -v and look for Metal buffer lines, or watch GPU History in Activity Monitor. We have not measured the forks on Metal; the TheTom fork reports turbo3 at 98.9% of q8_0 speed on an M5 Max (Part 3).

7. Community implementations#

The backend lists come from each fork's source tree, not from our runs; we measured only CUDA. In TheTom's tree, CUDA, Metal and SYCL have turbo flash-attention kernels. Vulkan has turbo2 to turbo4 dequantization shaders; we did not check how complete its flash-attention path is.

Wrapping up#

1. Mainline llama.cpp does not have TurboQuant. Build TheTom's or buun's fork, or use vLLM.
2. Start with -ctk turbo4 -ctv turbo4 -fa on.
3. Run llama-perplexity at your serving context against an f16 run before you trust it.

TurboQuant is one of twelve levers for cutting KV cache cost, and the twelve do not pay in the same unit. We measured all of them on the same A100 in KV Cache Reduction, Measured — Part 1.

Quantizing the cache is one approach; the other is changing what the attention layer stores in the first place. The new series starts there: DeepSeek's Config Says 16 KV Heads, Its Cache Holds a Ninth of That.

Want to run GPTQ, AWQ, GGUF and QLoRA yourself instead of reading our numbers? That is LLM Quantization and Compression Hands-On — 24 lectures, first 3 free.

New measurements go out weekly with Paper of the Week — subscribe below if you want the next one.