LLM Inference Optimization Part 1 — Attention Mechanism Deep Dive
Build Self-Attention from scratch. Compare MHA → GQA → MQA evolution in code. KV Cache mechanics and Prefill vs Decode analysis.
LLM Inference Optimization Part 1 — Attention Mechanism Deep Dive
When you deploy an LLM to a production service, the first wall you hit is inference speed and memory. No matter how good the model is, it's useless if it's slow and expensive. In this series, we dissect the core bottlenecks of LLM inference one by one and cover practical optimization techniques with code.
In Part 1, we implement the Attention mechanism from scratch — the starting point of all optimizations — and compare the evolution from MHA to GQA to MQA directly in code.
Self-Attention — Implementing from Scratch
Basic Structure
This part is for subscribers
A subscription unlocks every premium series and its Jupyter notebooks.
You need a free account to subscribe. Cancel anytime.
Related Posts
KV Cache Reduction, Measured on One A100 — Part 1: The Twelve Techniques Don't Pay in the Same Currency
Every list of KV cache techniques presents a dozen of them as peers. On one A100 the spread is enormous: prefix reuse cut a shared-prefix workload from 59.4s to 6.2s, while tuning the paged block size moved capacity by under 1%. The reason you cannot rank them on one number is that they pay out in different units.
llama.cpp KV Cache Quantization: Why q8_0 Costs 9% of Throughput — or 22%
Mainline llama.cpp on one A100, Qwen3-8B Q4_K_M, llama-server with 1 to 32 concurrent slots. On a 32K prompt, q8_0 cost 9% of server throughput when each request generated 128 tokens and 22% when it generated 1,024, because prefill dominates the short case and prefill is unaffected by the KV type. Per-token decode was 34% slower, matching llama-bench. VRAM in use after startup fell from 41.0 GiB to 25.1 GiB at four 64K slots.
llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K
Mainline llama.cpp, Qwen3-8B, one A100: -ctk q8_0 -ctv q8_0 matches f16 perplexity and cuts the 32K cache by 2.1 GiB, but decode at 64K depth drops to 55% of f16 (q4_0: 50%). Two other settings, q5_1 and a q8_0/q4_0 mix, silently ran prefill on the CPU at 43 and 63 tokens per second.