TurboQuant Status Check, August 2026 — What Actually Shipped in vLLM, llama.cpp, and Ollama

TurboQuant Status Check, August 2026 — What Actually Shipped in vLLM, llama.cpp, and Ollama

Five months after Google Research's TurboQuant paper landed, the "4x KV cache compression for free" story has split into three very different realities depending on which inference engine you run. One engine shipped it and published a sobering benchmark. One engine rejected it. One engine's implementation is dead.

This post is a status check — and a correction. Our earlier posts on TurboQuant repeated a claim that was circulating in March: that llama.cpp had merged tq1_0–tq4_0 cache types. That was wrong. Below is what the repositories actually say as of late August 2026, with links.

This is Part 3 of our TurboQuant series.

• Part 1: TurboQuant Explained — the algorithm (PolarQuant + QJL)
• Part 2: TurboQuant in Practice — llama.cpp forks and the Python package
• Part 3 (this post): What actually shipped, engine by engine
• Part 4: TurboQuant in vLLM on one A100 — measured (coming)

The Short Version

If you only remember one line: the only production engine with TurboQuant is vLLM, and vLLM's own recommendation is to keep using FP8 unless you specifically need the capacity.

1. vLLM: Shipped, Studied, and Honestly Reported

The one place TurboQuant is a first-class feature is vLLM. PR #38479 merged on April 15, 2026 and landed in v0.20.0. It adds a custom Triton-based TurboQuantAttentionBackend with four presets:

vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4      # FP8 keys, 4-bit values (~2.4x)
vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_4bit_nc   # 4-bit K+V, norm correction (~3.4x)
vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k3v4_nc   # 3-bit K, 4-bit V
vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_3bit_nc   # 3-bit K+V (~4x+)

Two details worth knowing before you try it:

• v0.20.0 supported only full-attention and uniform sliding-window transformers; hybrid Mamba/linear-attention models raised NotImplementedError. PR #39931 (merged May 5) added Qwen3.5 and Qwen3-Next by quantizing only the full_attention layers and skipping the linear ones — with an open edge case (#41726, chunked-prefill continuation) still being worked.
• Ampere GPUs (A100) needed a Triton float8 fallback; Hopper is the primary target.

What vLLM's own benchmark found

On May 11, the vLLM team published "A First Comprehensive Study of TurboQuant" — Llama-3.3-70B, Qwen3-30B-A3B (Instruct and Thinking), and MiniMax-M2.7, evaluated on MRCR long-context retrieval and reasoning benchmarks (AIME25, GPQA Diamond, MATH500, LiveCodeBench). The numbers are the most careful public evaluation to date, and they are not the numbers in the hype cycle.

Accuracy, relative to BF16:

On Qwen3-30B at 256K context, the 3-bit variants fell to 31–34% AUC — roughly a 30% relative degradation.

Speed, relative to BF16 (Qwen3-30B):

Capacity: FP8 2x, k8v4 2.4x, 4bit-nc 3.4x, 3bit-nc 4x+. Under burst load on Llama-70B, that capacity mattered: BF16 hit ~17s P99 TTFT from memory saturation while TurboQuant variants stayed under 3.5s. FP8 was best at 1.3s.

Their conclusion, quoted: *"FP8 remains the best default."* The practical variant is 4bit-nc — "up to 3.4x KV-cache capacity with modest accuracy degradation of 1–4 points" — and the 3-bit variants are explicitly not recommended for production.

What this means

The paper's headline was about distortion on isolated vectors. vLLM measured what happens to end-task accuracy and throughput in a real serving stack. The two are not the same thing: 3-bit keys cost you ~20 points on reasoning benchmarks, and the Triton kernels cost 10–60% latency against FP8's fused paths. TurboQuant in vLLM is a capacity tool for memory-bound long-context serving, not a free lunch.

2. llama.cpp: Rejected Upstream, Alive in Forks

This is where our earlier posts were wrong, so let's be precise.

Upstream ggml-org/llama.cpp has no TurboQuant cache type. The accepted -ctk/-ctv values in common/arg.cpp are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. No tq3_0, no turbo3.

What happened:

• PR #21010 (Vulkan TQ3_0) was closed for violating the project's AI-generated-code policy.
• PR #21089 added CPU-only tbq3_0 / tbq4_0 types. Reported on Qwen3.5-4B: tbq4_0 3.94x compression with KLD similar to q4_0; tbq3_0 5.22x with expected quality loss; CPU decode fell from 15.7 to ~8 tok/s. Closed June 2, 2026. The maintainers' reasoning: Hadamard rotations are already merged, and the PR did not demonstrate gains over the existing equal-bit-width types across multiple models, as the contribution guidelines require.

That rejection is defensible. The upstream concern isn't "TurboQuant doesn't work" — it's "show us it beats q4_0 + Hadamard on the same bits, on more than one model." Nobody has yet done that to the maintainers' standard.

Meanwhile the community discussion (#20969) produced a genuinely impressive set of independent implementations:

And they converged on the same engineering conclusions — this is the most useful technical output of the whole episode:

1. Algorithm 1 (PolarQuant MSE) alone is enough. The QJL residual stage adds overhead without measurable benefit. Every fork dropped it.
2. Keys need more bits than values. K/V norm disparities of up to 182x were measured across models; asymmetric k4/v3-style allocation is the norm now (vLLM's k8v4 preset is the same lesson).
3. Block size 32 beats the paper's 128 for flash-attention parallelism.
4. Norm correction (store ‖x‖ / ‖Q(x)‖ and rescale at decode) improves perplexity at zero cost. It's the _nc suffix in vLLM.

If you run llama.cpp on your own hardware, a fork is a perfectly good way to get 4–5x KV compression today. Just know that you're on a branch that tracks upstream by hand.

3. Ollama: Dead on Arrival

Ollama PR #15505 was the most complete implementation of the lot — nine OLLAMA_KV_CACHE_TYPE presets (tq4/tq3/tq2, K-only and V-only variants), FWHT rotation, Lloyd–Max codebook, outlier splitting, and a fused inline-decode flash-attention path. On Blackwell, tq3k reportedly held ~80% of f16 memory at 60–75% of f16 decode throughput.

It was closed May 30, 2026, by the author, because Ollama's #16031 moved GGUF inference to an external upstream llama-server. Ollama no longer has its own attention path to patch. TurboQuant in Ollama now depends on TurboQuant in upstream llama.cpp — which, see above, was rejected.

So the honest answer to "does Ollama support TurboQuant?" is: no, and not until llama.cpp upstream does.

4. HuggingFace: An Independent Package, Not Google's

pip install turboquant works and gives you TurboQuantCache, a drop-in past_key_values for Transformers, plus a TurboQuantMSE quantizer and an OpenAI-compatible server. It's published by back2matching, Apache-2.0, versions 0.1.0 and 0.2.0 released March 25–27, 2026.

Its README says it plainly: *"This is an independent implementation, not affiliated with Google Research."* Google has not released an official implementation. Treat the package as what it is — a well-made community reimplementation, good for experiments, not a reference.

5. What the Paper Claimed vs. What Shipped

6. So What Should You Do?

• Serving with vLLM, memory-bound on long context: try turboquant_4bit_nc, measure *your* task, keep fp8 as the control. Skip the 3-bit variants.
• Serving with vLLM, not memory-bound: fp8. Done.
• llama.cpp on a Mac or a single GPU: a fork (turbo3/turbo4 with -fa on) is real and fast. Use turbo4 for keys if quality matters.
• Ollama: nothing to do. Wait for upstream.
• Research / notebooks: pip install turboquant, or implement PolarQuant yourself — it's ~60 lines (Part 5 of this series does exactly that on real KV tensors).

In Part 4 we stop reading other people's tables and run vLLM's four presets against FP8 and BF16 on a single A100 with an 8B model — the size vLLM's study didn't cover.

References

• Zandieh et al., TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate, ICLR 2026
• vLLM, A First Comprehensive Study of TurboQuant: Accuracy and Performance (May 11, 2026)
• vLLM PR #38479 (merged Apr 15, 2026) · PR #39931 hybrid models (merged May 5) · Issue #41726
• llama.cpp PR #21089 (closed Jun 2, 2026) · PR #21010 · Discussion #20969
• Ollama PR #15505 (closed May 30, 2026)
• turboquant on PyPI · back2matching/turboquant