Models & Algorithms•SOTAAZ Lab••KR

llama.cpp KV Cache Quantization: Which -ctk and -ctv to Use, Measured on the Current Build

The -ctk/-ctv (--cache-type-k/-v) settings to copy, the allowed types, and the GGML_CUDA_FA_QUANTS build option. Measured: q8_0 saved 4.2 GiB at 64K and decoded at 56% of f16's speed.

llama.cpp KV Cache Quantization: Which -ctk and -ctv to Use, Measured on the Current Build

Short answer. If the KV cache fits in VRAM, leave it at the default f16. If it does not, use -ctk q8_0 -ctv q8_0. Set both flags to the same value, and do not expect the memory to be free: at 64K tokens of context the quantized cache decodes at roughly half the speed of f16.

Decision path. Does the f16 KV cache fit in VRAM? Yes: leave it at f16, which decoded 79.6 tokens per second at 64K depth. No: set -ctk q8_0 -ctv q8_0, perplexity within 0.1% of f16 but 56% of f16 decode speed at 64K. Still does not fit: -ctk q4_0 -ctv q4_0, 1.1 GiB less than q8_0 at 32K and 0.5 to 1.1% higher perplexity. Skip q5_1, the q8_0/q4_0 mix and iq4_nl, which were slower than q8_0. Qwen3-8B Q4_K_M on one A100.

This page is the short version of three measurement posts on this blog, rechecked on the llama.cpp build of 4 October 2026. The recheck mattered. A change merged on 9 September altered what happens with the less common types, and part of what we published in September no longer holds on current builds. Where a number below comes from an older build, it says so.

The same results in 97 seconds.Subscribe to SOTAAZ on YouTube

The commands

bash
# server, chat, CLI: the same two flags
llama-server -m model.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 32768

# compare types on your own card before you commit to one, one type per run
for t in f16 q8_0 q4_0; do
  llama-bench -m model.gguf -ngl 99 -fa on -ctk $t -ctv $t -p 4096 -n 128 -d 0 -d 32768
done

-ctk and -ctv are the short forms of --cache-type-k and --cache-type-v. Keep -fa on: quantized V caches need flash attention in llama.cpp. llama-bench also accepts comma-separated lists, but -ctk f16,q8_0,q4_0 -ctv f16,q8_0,q4_0 runs all nine K/V combinations, six of them the mixed pairs this page advises against. The loop keeps K and V equal.

K cacheV cache
Short flag-ctk-ctv
Long flag--cache-type-k--cache-type-v
Environment variableLLAMA_ARG_CACHE_TYPE_KLLAMA_ARG_CACHE_TYPE_V

Accepted values for both: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. The default is f16. Anything else stops with Unsupported cache type.

With -fa auto, the default, llama.cpp turns flash attention on for you when the V cache is quantized and logs enabling flash_attn since it is required for quantized V cache; with -fa off it refuses to start.

If you see error: invalid argument: --ctk, the short flag got a double dash. There is no --ctk. Use -ctk q8_0 or the long form --cache-type-k q8_0.

Starting from nothing? Episode 1 of our how-to series goes from an empty folder to a CUDA build and a first answer. That build gives you one llama program, so llama-server and llama-bench here become llama serve and llama bench. Both list the same -ctk and -ctv flags.

A 3 by 3 grid of -ctk (K cache) against -ctv (V cache) for f16, q8_0 and q4_0. The three equal pairs decoded at 79.6, 44.3 and 42.4 tokens per second at 64K depth. The six mixed pairs are the ones this page advises against; the one we measured, K q8_0 with V q4_0, decoded at 33.8, slower than both q8_0 and q4_0. Qwen3-8B Q4_K_M, one A100, llama.cpp b11390.

What each setting costs

Qwen3-8B Q4_K_M on one A100 80GB. Speeds are tokens per second on the 4 October build (b11390 (dd26678), default CUDA options). Memory is the cache alone at 32K tokens, computed from the type's bits. Perplexity is from the September build (b10753 (69320fe)); the cache format is set by the type, so it should not move with the kernels, but we did not re-run it.

-ctk / -ctvCache at 32KPerplexity vs f16 (4K–32K)Prefill 4096Decode at 64K depth
f16 / f164.50 GiB—4,70479.6 (100%)
q8_0 / q8_02.39 GiB−0.0% to +0.1%4,58444.3 (56%)
q4_0 / q4_01.27 GiB+0.5% to +1.1%4,59942.4 (53%)
bf16 / bf164.50 GiBnot measured4,67241.3 (52%)
q5_1 / q5_11.69 GiBnot measured4,58327.8 (35%)
q8_0 / q4_01.83 GiBnot measured4,58233.8 (42%)
iq4_nl / iq4_nl1.27 GiBnot measured104not measured

Three things in that table decide most setups.

Prefill barely moves; decode at depth does. Every working type prefills within 3% of f16. The cost shows up when the cache is already long, because each new token reads all of it. With 8K tokens in the cache, q8_0 decoded at 83% of f16; at 64K, 56%. The memory you saved is paid back in time at exactly the context length where you needed the memory.

Animation: the K and V cache gains one row per generated token, and before each new row is added, the new token reads every row already cached. Prefill pays once; every decoded token pays again for the whole cache. Measured on Qwen3-8B with one A100: q8_0 decoded at 83% of f16 speed with 8K tokens cached and 56% with 64K.

q8_0 and q4_0 decode at nearly the same speed, within 5% of each other at 8K, 32K and 64K. Between them the choice is memory against quality: q4_0 holds about 1.1 GiB less at 32K for this model and costs about 1% perplexity. Take q4_0 only when that gigabyte decides whether the model loads.

For a case where q4_0 is what gets a model inside the budget, and what happened when we then gave that setup coding tasks, see We Forced Qwen3.8 27B Into 8GB and the video.

q4_0 getting Qwen3.8 27B inside an 8 GB budget on an A100, then the coding tasks.Subscribe to SOTAAZ on YouTube

The memory saving is real. Peak VRAM during the 64K runs fell by 4.19 GiB with q8_0 and 6.45 GiB with q4_0. The arithmetic predicts 4.22 and 6.47, so the measured peak matched it to within 0.03 GiB.

Bar chart of peak VRAM saved against f16 during the 64K runs. q8_0: predicted 4.22 GiB, measured 4.19 GiB. q4_0: predicted 6.47 GiB, measured 6.45 GiB. The prediction is the f16 cache minus the new type's cache at 64K tokens; both gaps are within 0.03 GiB.

Types that work but you should not pick

On the September build, four of these types and every mixed K/V pair prefilled at 83 to 284 tokens per second against about 4,600, because the default CUDA build compiled no flash-attention kernel for them. We published that as "set both flags to the same value, and use only four types."

On the current build that cliff is gone for most of them. A change merged on 9 September (llama.cpp #28079) did two things. It made q4_1, q5_0 and q5_1 supported types for flash attention. And when the decode kernel for a K/V pair was not compiled, it now converts K and V to f16 on the fly instead of failing over. llama-server prints one line when that happens:

ggml_cuda_flash_attn_ext_vec: no FlashAttention vector kernel compiled for K/V types q5_1-q5_1, converting K and V to f16 instead (slow). Add "q5_1-q5_1" to GGML_CUDA_FA_QUANTS to compile it.

llama-bench does not show that line unless you pass -v, because it silences the library's log by default.

Animation of -ctk q5_1 -ctv q5_1 on the default CUDA build. On the September build there was no flash-attention kernel and these types prefilled at 83 to 284 tokens per second against about 4,600 for f16. On the current build (b11390, after llama.cpp #28079) q5_1 is a flash-attention type and prefills at 4,583 tokens per second against 4,704, but with no decode kernel compiled it converts K and V to f16 each step and decodes at 27.8 tokens per second at 64K, 35% of f16. Adding q5_1-q5_1 to GGML_CUDA_FA_QUANTS compiles the kernel; we did not measure that build.

So q5_1 and the q8_0 / q4_0 mix now prefill at full speed, and each gave a normal answer to one short test request on llama-server; we did not measure their quality. They are still a bad trade on a default build. q5_1 decodes at 35% of f16 at 64K, slower than q4_0, while holding more memory than q4_0. The mix sits between them on speed and saves less than q4_0. bf16 is the other trap: it holds exactly as much as f16 and decodes at 52% of it at 64K, so it buys nothing on this card.

Scatter plot of cache size at 32K tokens against decode speed at 64K depth. Kept types: f16 4.50 GiB and 79.6 tokens per second, q8_0 2.39 GiB and 44.3, q4_0 1.27 GiB and 42.4. Skipped types: bf16 4.50 GiB and 41.3 (same memory as f16, about half the speed), q5_1 1.69 GiB and 27.8, the q8_0/q4_0 mix 1.83 GiB and 33.8 (both use more memory than q4_0 and are slower). iq4_nl is not shown: it prefilled at 104 tokens per second and its decode was not measured.

iq4_nl is the one type that is still unusable. -ctk and -ctv accept it, but no build can compile a CUDA flash-attention kernel for it, because it is not an allowed type in GGML_CUDA_FA_QUANTS. It prefilled at 104 tokens per second.

Build b12274 later added a new type, f8_e4m3. On the same A100 its cache was 6% smaller than q8_0, but it moved the output about 8 to 10 times as far from f16 as q8_0 did, and decoded 12-13% slower at 32K. The full results are in llama.cpp's new FP8 KV cache vs q8_0.

Build option: GGML_CUDA_FA_QUANTS (replaces GGML_CUDA_FA_ALL_QUANTS)

If you have a reason to want q5_1 or a mixed pair, compile its decode kernel in. These are the build commands, with q5_1 added to the default list:

bash
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16;q5_1-q5_1"
cmake --build build --config Release -j

-DGGML_CUDA_FA_QUANTS=all compiles every pair, at the cost of a much longer build. The old option GGML_CUDA_FA_ALL_QUANTS is now a deprecated alias for all. Types allowed in the list: f16, bf16, q4_0, q4_1, q5_0, q5_1, q8_0. We have not measured a build with the extra kernels on the current commit.

On a server, output length decides the cost

Single-stream decode is the worst case. In the llama-server measurement on the September build, with a 32K prompt, q8_0 cost 9% of total throughput when each request produced 128 tokens and 22% when it produced 1,024. Prefill takes most of a short-answer request and quantization barely touches prefill. If your answers are long relative to your prompts, measure at your own ratio.

Animation of one llama-server request with a 32K-token prompt: the prefill part stays fixed and quantization barely touches it, while the decode part, which reads the whole cache for every token, grows from 128 to 1,024 tokens. Measured on the September build: q8_0 cost 9% of total throughput at 128 output tokens and 22% at 1,024. Bar widths are a sketch, not to scale.

Before you trust a number from your own card

  • Check that nothing else is on the GPU. nvidia-smi --query-compute-apps=pid,used_memory --format=csv. Our first sweep in September read f16 prefill at 1,976 tokens per second instead of 4,707, because a run we thought we had stopped was still sharing the card.
  • Measure at the depth you will run. At an empty cache every working type looks the same. Add -d 32768 (or your real context) to llama-bench.
  • Watch the log, not just the speed. The converting K and V to f16 line is the only sign that a pair fell off its compiled kernel.

How big the cache is for your model

Per token, the cache holds layers × KV heads × head dimension × 2 (K and V) × bytes per value. Qwen3-8B has 36 layers and 8 KV heads of dimension 128, which is 144 KiB per token at f16, or 4.5 GiB at 32K. q8_0 stores 8.5 bits per value and q4_0 4.5 bits. A model with more KV heads has a proportionally larger cache and, plausibly, a larger slowdown at depth; we have measured only this one. The general formula and a script for other models are in KV Cache Explained.

The KV cache size formula M = 2 × L × H_kv × d_head × n × b, worked for Qwen3-8B: 2 × 36 layers × 8 KV heads × 128 dims × 32,768 tokens × b. That is 144 KiB per token at f16 and 4.50 GiB for 32K tokens. Only b is set by -ctk/-ctv: f16 at 2 bytes gives 4.50 GiB, q8_0 at 8.5 bits gives 2.39 GiB, q4_0 at 4.5 bits gives 1.27 GiB.

This formula assumes every layer keeps full K and V for each KV head. Newer models break that assumption in different ways, and we have since measured six of them: Four Times the Context, Less Memory.

What this does not show

One model, one weight quantization, one A100, batch size one for the speed table. Perplexity and the server numbers come from the September build, not the current one. The 64K decode figures are a single run each. llama.cpp changes weekly, and the 9 September change is a reminder that a table like this one can go stale in a month; the date and commit are in the note below so you can tell.

The three posts behind this page have the full tables: single-stream speed, memory and perplexity, llama-server under concurrent load, and every -ctk value on the September build. The sweep scripts are in the free KV cache benchmark harness.

Current-build measurements: llama.cpp b11390 (dd26678) (ggml-org master, 4 October 2026), CUDA build for sm80 with default options (GGML_CUDA_FA_QUANTS=q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16), Qwen3-8B Q4_K_M, one A100 80GB PCIe, -fa on -ngl 99. Prefill: -p 4096 -n 128 -d 0 -d 8192 -r 2. Depth: -p 0 -n 128 -d 32768 -d 65536 -r 1, VRAM sampled every 0.5 s with nvidia-smi. Warning lines from llama-server with -c 8192 and one short request. Scripts scripts/bench-kv-types-master.sh and scripts/bench-kv-depth-master.sh; raw JSON and logs in drafts/llamacpp-kv-guide/. Perplexity and server figures: llama.cpp b10753 (69320fe) (2 September 2026), same card and model, from the posts linked above.