TurboQuant in vLLM on One A100 — Capacity, Speed, and Accuracy of All Four Presets on an 8B Model

TurboQuant in vLLM on One A100 — Capacity, Speed, and Accuracy of All Four Presets on an 8B Model

vLLM is the only production engine that ships TurboQuant, and vLLM's own study (Part 3) covered 30B–200B models on H100s. That leaves the question most single-GPU users actually have: what do the presets do on one A100, with an 8B model — the setup where you'd be tempted to trade quality for capacity?

We measured all six KV dtypes — bf16, fp8, and the four TurboQuant presets — on Qwen3-8B (bf16 weights, max context 40,960) with vLLM 0.28.0 (+cu129 wheel) on an A100 80GB. Per dtype: KV capacity, batched throughput (64 × 1K→512), single-stream TTFT and decode at 32K context, needle-in-a-haystack at 4K/16K/32K (8 samples each), and GSM8K (200 questions, no-think, greedy).

Three results, one of them a bug report:

1. Capacity is real: up to 3.5x. turboquant_3bit_nc fits 1.37M cache tokens where bf16 fits 396K.
2. On Ampere, every TurboQuant preset costs about half your throughput. 1,700–2,160 tok/s versus 3,300–3,450 for bf16/fp8, and a third of the decode speed at 32K. The Triton path that is "10–20% overhead" on Hopper is much heavier on sm80.
3. turboquant_k8v4 is broken on A100. GSM8K collapses from 91% to 3.5% and needle recall to 1/24 — not noise, but visibly degenerated output. The other three presets are accurate.

1. Capacity: the Part That Works Exactly as Advertised

!KV capacity by dtype

These come straight from vLLM's allocator log at gpu_memory_utilization=0.90. If your problem is "how many long-context users fit on this GPU," the presets deliver what the bit math promises.

2. Speed: on Ampere, the Toll Is Heavy

!Throughput and 32K decode

TTFT barely moves — prefill is compute-bound and the quantized store is cheap. Everything else is expensive: batched throughput halves, and single-stream decode at 32K drops to about a third of bf16. vLLM's study measured 73–80% of baseline throughput on Hopper; on Ampere, where the Triton kernels run through the float8e4b15 fallback and without Hopper's fp8 tensor cores, the same presets land at 51–65%. The PR's own caveat — "Hopper is the primary target" — is worth taking literally.

Note fp8 is faster than bf16 here (+3%) while doubling capacity. On an A100, fp8 isn't a compromise; it's a straight upgrade.

3. Accuracy: Three Presets Fine, One Broken

!Accuracy by dtype

Two things here deserve care.

First, the k8v4 failure is real, and I initially blamed my own harness. When GSM8K came back at 3.5%, my first theory was a broken answer parser or a mangled chat template, so I ran 4bit_nc through the identical harness: 91%. Not the harness. Opening the k8v4 answers settled it — greedy decoding, and this is what came back:

Alright, the first and then the final answer.

Alright, the first and then the final answer.

Alright, the first and then the final answer.
...

Short answers still pass, by the way — "What is the capital of France?" gets "Paris." — which is exactly why a smoke test won't catch this. A ~3K-token needle prompt answers with the needle corrupted and the filler leaking: '12345678956789. The quick brown fox jumps over the...'. The k8v4 preset stores keys as fp8 — on Ampere that path uses a Triton float8 fallback (tl.float8e4b15), and the symptoms (fine at short range, degenerate as quantized keys accumulate) point there. vLLM's PR noted early quality failures on tq3 that were fixed by K/V rebalancing; this looks like the same class of defect surviving on sm80. If you are on Ampere, do not use turboquant_k8v4 without running your own eval first. The other presets showed nothing like this, and vLLM's Hopper-based study reported k8v4 as its safest preset — which is exactly why the Ampere check was worth doing.

Second, our 3-bit numbers look better than vLLM's — understand why before celebrating. Their −20-point drops were on AIME25 and LiveCodeBench with long thinking traces on bigger models at up to 256K context; our GSM8K is short-form arithmetic at 4K with no-think decoding, and NIAH is retrieval, which the _nc presets handle well (Part 6 shows why norm correction specifically fixes attention retrieval). Easy tasks tolerate 3-bit keys; hard reasoning does not. Both measurements are right; they bracket the range.

4. So, on an A100, When Would You Use Any of This?

The one-sentence version: on Ampere, TurboQuant in vLLM is a capacity tool that charges half your throughput as toll, and one of its four presets is currently broken. On Hopper, trust vLLM's own study instead — the kernels were written for it.

Harness: bench-vllm-tq.py (attached). Each dtype in a fresh process, greedy decoding throughout.

Hardware: 1× A100 80GB, driver 535 / CUDA 12.2. Software: vLLM 0.28.0 (+cu129), Qwen3-8B bf16, max context 40,960.

Wall-clock: ~70 min for all six dtypes. Raw results: vllm-tq.json in the bundle.

Verified on: 2026-08-31.

References

• Part 3: TurboQuant Status Check · Part 5: llama.cpp fork · Part 6: From Scratch
• vLLM, A First Comprehensive Study of TurboQuant · PR #38479
• Zandieh et al., TurboQuant, ICLR 2026