llama.cpp -ctk and -ctv: The Default CUDA Build Supports Four Values

llama.cpp `-ctk` and `-ctv`: The Default CUDA Build Supports Four Values

-ctk and -ctv set the data type of the KV cache. llama-bench's help prints the default and nothing else; llama-server's does list the nine allowed values, but neither says what any of them costs. I measured every accepted value on one A100.

Short answer: the accepted list depends on which binary you run, and the default CUDA build compiles FlashAttention kernels for only four of the values — and none at all when K and V differ. That is decided at compile time, so nothing warns you at runtime.

The two lists are not the same

llama-bench parses these flags itself, in tools/llama-bench/llama-bench.cpp, where ggml_type_from_name knows eight names. Everything else goes through common/arg.cpp, whose kv_cache_types vector holds nine — the same eight plus f32. So llama-server -ctk f32 starts and llama-bench -ctk f32 exits with error: invalid parameter for argument: -ctk. Nothing in the help output says so.

Both flags also take a comma-separated list in llama-bench, so one command sweeps them:

llama-bench -m model.gguf -fa on -ngl 99 -ctk f16,q8_0,q4_0 -ctv f16,q8_0,q4_0 -p 4096 -n 128 -d 0 -d 8192

Four of the eight fall off the CUDA FlashAttention path

Qwen3-8B Q4_K_M on one A100 80GB, -fa on -ngl 99, K and V set to the same type:

All figures are tokens per second, two repetitions each. The empty @32K cells are configurations that never reached that depth inside the 15-minute per-configuration limit. The @8K column shows why: prefilling 4,096 tokens on top of an 8K cache at 18.7 tokens per second takes 3 minutes 39 seconds.

The gap between the two groups is not a tax, it is a cliff. The four slow types hold 1.27 to 1.69 GiB of cache at 32K and prefill 33 to 55 times slower than q4_0. iq4_nl and q4_0 store exactly the same 40.5 KiB per token, and one is 52 times faster to fill. Generation collapses too once depth accumulates: at 8K the slow four decode at 6.1 to 7.2 tokens per second against 107.2 for q4_0.

The cause is a build option

No guessing needed this time. ggml/src/ggml-cuda/fattn.cu checks two things before it picks a kernel:

#ifndef GGML_CUDA_FA_ALL_QUANTS
    if (K->type != V->type) {
        return BEST_FATTN_KERNEL_NONE;
    }
#endif

and ggml_cuda_fattn_kv_type_supported in the same file returns true for f32, f16, bf16, q8_0 and q4_0, while q4_1, q5_0 and q5_1 return true only when GGML_CUDA_FA_ALL_QUANTS is defined. iq4_nl is in neither list, so it is always false.

That option defaults to OFF in ggml/CMakeLists.txt, and my build's CMakeCache.txt confirms OFF. So the four slow types and every mismatched pair have no FlashAttention kernel compiled at all: kernel selection returns BEST_FATTN_KERNEL_NONE and the work falls to the slow general path, -fa on notwithstanding.

This is a build configuration, not a limit of the card. If you need those combinations, rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON and pay for it in compile time and binary size. iq4_nl stays unsupported either way. The next section measures how much comes back.

Mixing K and V breaks it too

-ctk and -ctv are separate flags, so nothing stops you from quantizing the V cache harder than the K cache. On this build, any mismatch drops off the fast path, including pairs where both types are fast alone:

In the default build the rule is simple: set both flags to the same value. An earlier post here reported the q8_0 / q4_0 mix as slow and treated it as a bad pair. It is not the pair. It is the mismatch.

How much the option brings back

If the cause is a build option, turn it on. I rebuilt the same commit with -DGGML_CUDA_FA_ALL_QUANTS=ON and measured under the same conditions. Both builds are CMAKE_BUILD_TYPE=Release, sm80.

The two control rows agree to the decimal, which is what makes the rest comparable. With the option on, the three slow types and the mismatched pair all run at full speed: 4,582 to 4,593 tokens per second of prefill against 4,599 for q4_0.

iq4_nl does not move, exactly as the source predicts. There is no path to using it as a KV cache type at this commit.

The price is compile time and binary size, since the build emits many more kernel combinations. Unless you specifically need q5_1's footprint (1.69 GiB at 32K), staying on the default build with q8_0 or q4_0 is the simpler answer.

bf16 is the one that looks safe and is not

bf16 costs the same memory as f16, 144 KiB per token, and keeps the fast prefill path. It looks like a free swap. At depth it is the worst of the four working types: 44.3 tokens per second at 32K against 104.6 for f16, and below both quantized types. If you are holding 4.5 GiB of cache anyway, f16 decodes 2.4 times faster at that depth.

What to set

• The cache fits in VRAM. Keep f16. It is the fastest at every depth measured.
• You need the memory back. -ctk q8_0 -ctv q8_0 halves the cache to 2.39 GiB at 32K and costs 36% of decode throughput at that depth. q4_0 gives 1.27 GiB and costs 39%.
• Anything else in the list, or a K/V mismatch. The default build has no FlashAttention kernel for it. Rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON and measure before shipping. The failure is silent: the model loads, answers correctly, and feeds a long prompt at a walking pace.

The memory column is arithmetic, not a measurement: 36 layers × 8 KV heads × 128 dims × 2 (K and V) × bytes per element, times the context length. Qwen3-8B uses grouped-query attention with 8 KV heads, which is why 32K of f16 cache is 4.5 GiB rather than tens of gigabytes.

The trap that cost me this table once

The first sweep produced f16 prefill at 1,976 tokens per second. The real figure is 4,707. A second benchmark process was still on the same GPU from a run I thought I had stopped, and the two shared the card. Nothing in the output says so — llama-bench reports a clean mean with a small standard deviation, because both processes were slowed steadily.

If you benchmark on a shared box, check nvidia-smi --query-compute-apps=pid,used_memory --format=csv before you trust a number, and run one sweep at a time. The sweep script in the harness kit now refuses to start if another llama-bench is alive — though that only guards against my own runs, not someone else's process on a shared card.

Harness and raw results: the sweep script and the per-configuration JSON ship with the free KV cache benchmark harness kit.

Setup: llama.cpp 69320fe (CUDA, sm80, CMAKE_BUILD_TYPE=Release), Qwen3-8B Q4_K_M, one A100 80GB PCIe. The main tables use the default build with GGML_CUDA_FA_ALL_QUANTS=OFF; the comparison section rebuilds the same commit with it ON and measures on the same card. Settings: -fa on -ngl 99 -p 4096 -n 128 -d 0 -d 8192 -d 32768 -r 2, one process at a time; the mixed K/V pairs were run at depth 0 with -r 1. Accepted-value lists read from tools/llama-bench/llama-bench.cpp (ggml_type_from_name) and common/arg.cpp (kv_cache_types) at the same commit. Measured on 2026-09-21.