AI Engineering••KR

LLM Inference Optimization Part 3 — Sparse Attention in Practice

Sliding Window, Sink Attention, DeepSeek DSA, IndexCache, and Nvidia DMS. From dynamic token selection to Needle-in-a-Haystack evaluation.

LLM Inference Optimization Part 3 — Sparse Attention in Practice

LLM Inference Optimization Part 3 — Sparse Attention in Practice

In Part 2, we covered KV Cache quantization, compression, and PagedAttention. Those techniques focus on reducing stored data. Part 3 shifts direction and tackles reducing the computation itself with Sparse Attention.

The key question: "Do we really need every token?"

In most cases, the answer is no. In a 128K context, the tokens that the current token actually needs to attend to are only 5–20% of the total.

The Problem with Full Attention

🔐

This part is for subscribers

A subscription unlocks every premium series and its Jupyter notebooks.

You need a free account to subscribe. Cancel anytime.

Related Posts

KV Cache Reduction, Measured on One A100 — Part 1: The Twelve Techniques Don't Pay in the Same Currency
Models & Algorithms

KV Cache Reduction, Measured on One A100 — Part 1: The Twelve Techniques Don't Pay in the Same Currency

Every list of KV cache techniques presents a dozen of them as peers. On one A100 the spread is enormous: prefix reuse cut a shared-prefix workload from 59.4s to 6.2s, while tuning the paged block size moved capacity by under 1%. The reason you cannot rank them on one number is that they pay out in different units.

llama.cpp KV Cache Quantization: Why q8_0 Costs 9% of Throughput — or 22%
Models & Algorithms

llama.cpp KV Cache Quantization: Why q8_0 Costs 9% of Throughput — or 22%

Mainline llama.cpp on one A100, Qwen3-8B Q4_K_M, llama-server with 1 to 32 concurrent slots. On a 32K prompt, q8_0 cost 9% of server throughput when each request generated 128 tokens and 22% when it generated 1,024, because prefill dominates the short case and prefill is unaffected by the KV type. Per-token decode was 34% slower, matching llama-bench. VRAM in use after startup fell from 41.0 GiB to 25.1 GiB at four 64K slots.

llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K
Models & Algorithms

llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K

Mainline llama.cpp, Qwen3-8B, one A100: -ctk q8_0 -ctv q8_0 matches f16 perplexity and cuts the 32K cache by 2.1 GiB, but decode at 64K depth drops to 55% of f16 (q4_0: 50%). Two other settings, q5_1 and a q8_0/q4_0 mix, silently ran prefill on the CPU at 43 and 63 tokens per second.