llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K

llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K

Search Console says most people who land on TurboQuant in Practice were not looking for TurboQuant. They typed llama.cpp kv cache quantization, llama.cpp ctk ctv, cache-type-k. They want the thing that ships in mainline llama.cpp today: -ctk q8_0 -ctv q8_0, or q4_0, behind -fa on. That post is about a fork that upstream never merged, so it is the wrong answer to the right question.

This is the right answer, measured. Same card, same model, same four measurements I ran on the fork, so the two sit in one table at the end.

Rig

One A100 80GB PCIe, driver 535. llama.cpp mainline, ggml-org master 69320fe (2026-09-02), built with -DGGML_CUDA=ON for sm80. Qwen3-8B at Q4_K_M (4.68 GiB of weights). Every run uses -fa 1 -ngl 99; quantized V caches require flash attention in llama.cpp, so there is no non-FA comparison to make. llama-bench numbers are two-run averages. Perplexity is wikitext-2, 40 chunks at 4K context and 3 chunks at 8K, 16K and 32K.

The KV types are f16 (the default), q8_0, q4_0, q5_1, and the K q8_0 / V q4_0 mix that forum threads recommend. Two of those five did not survive contact with the hardware, and that is the first result.

Two of the five settings run prefill on the CPU

q8_0 and q4_0 cost three to four percent on prefill and six percent on generation. q5_1 prefills 8K tokens at 43 tokens per second — a hundred times slower than f16 — and the q8_0/q4_0 mix at 63. The q5_1 run alone took 2 hours 41 minutes for a test that takes the others a minute. I cut the mix short after its first row.

To see what was happening I reran short 2K prefills on the second, idle A100 while sampling GPU utilization every second. With q4_0, utilization sat at 72–100%. With q5_1 it sat at 2–4%, and with the mix at 4–25%. The GPU was idle; the attention op was running on the CPU. llama-bench prints no warning when this happens, and generation still runs on the GPU (q5_1 generates at 91.6 tok/s), so an interactive session feels merely sluggish rather than broken. You only notice when a long prompt takes minutes.

I looked for the reason in ggml-cuda/fattn.cu. The batched flash-attention kernels used for prefill work on f16 K/V and dequantize the cache on the way in; the single-token vector kernels have native paths for every quantized type. What decides that q8_0/q8_0 and q4_0/q4_0 get the fast batched path while q5_1/q5_1 and q8_0/q4_0 fall back, I could not pin down in the time I had, and I am not going to guess. The measurement stands on its own: on this build and this card, those two settings are not usable for anything with a real prompt.

Everything below is the three settings that work.

Memory: the deltas are exactly the nominal bits

Peak VRAM at 32K context, sampled with nvidia-smi during one perplexity chunk:

The peak includes 4.68 GiB of weights and the compute buffers, so the differences are the cache. The f16 cache for 32,768 tokens computes to 36 layers × 8 KV heads × 128 × (K+V) × 2 bytes × 32,768 = 4.50 GiB. q8_0 stores 8.5 bits per value, which predicts 2.39 GiB and a 2.11 GiB saving; q4_0 stores 4.5 bits, predicting 1.27 GiB and a 3.23 GiB saving. Both measured deltas match to the hundredth. These are the same numbers the fork produced for the same types, which is what you would expect — the cache layout is decided by the type, not the kernels.

Quality: q8_0 is lossless, q4_0 costs about one percent

q8_0 is inside the error bar at every context length. q4_0 sits 0.5–1.1% above f16 and does not drift with context, which is the property that separates it from the fork's turbo3 (fine at 4K, +210% at 16K). Three chunks at 8K and above is a thin sample; the ± column is doing real work in that table.

Speed at depth: this is the number that moves

Prefill and empty-cache generation barely change. The number that changes is decode when the cache is already full, because that is when every step reads the whole cache:

With 8K tokens in the cache, quantized KV costs about 20%. With 64K in the cache, it costs half. Reading a quantized cache means dequantizing it on every decode step, and that work grows with the context while the f16 path just streams bytes. The saving that made the long context fit is paid back, in time, at exactly the context length where you needed it.

The fork post measured the same two types at 27–29% of f16 at 64K. Mainline is at 50–55%: the quantized-KV decode path got roughly twice as fast between the fork's base and this commit. It is still a halving.

What to set

The choice between q8_0 and q4_0 turns out to be about memory alone: they decode at the same speed at every depth, so q4_0 is worth it only when the extra 1.1 GiB is the difference between fitting and not.

If you want to build these quantizers rather than flip their flags — GPTQ, AWQ, GGUF, and the KV-cache formats — that is what LLM Quantization and Compression covers, 24 lectures with the first three free.

What this does not show

One model at one weight quantization. Qwen3-8B uses 4:1 GQA with 8 KV heads; a model with more KV heads has a bigger cache and, presumably, a bigger depth penalty. Batch size one throughout — llama-bench decode is a single stream, and batched serving changes the bandwidth picture. Perplexity at 8K and above is three chunks. And the CPU-fallback finding is a fact about commit 69320fe on sm80; a later commit or a Hopper card may behave differently, and I have not checked either.

The fork comparison is in the TurboQuant llama.cpp post; the broader question of which KV technique pays in which unit is Part 1 of the KV series, where this post becomes the quantization row.

New measurements go out weekly with Paper of the Week — subscribe below if you want the next one.

Rig: A100 80GB PCIe × 1, driver 535, second A100 used only for the utilization probe. llama.cpp 69320fe (ggml-org master, 2026-09-02), CUDA build for sm80. Qwen3-8B Q4_K_M, -fa 1 -ngl 99. Harness scripts/bench-llamacpp-main.sh; raw JSON and logs in drafts/llamacpp-kv-bench/; tables produced by scripts/parse-llamacpp-kv-bench.py. Verified on 2026-09-15.