llama.cpp vs vLLM: Deployment Methods and Strategies, Part 2: Speed and Concurrent Requests

llama.cpp vs vLLM: Deployment Methods and Strategies, Part 2: Speed and Concurrent Requests

Part 1 ended on a trade. On Gemma 3 4B, vLLM reserves 2,682 MiB for one 32K request at max_num_batched_tokens 8,192, and 882 MiB at 512. A smaller value means vLLM reads a prompt in smaller pieces, so we expected that saving to cost prompt speed. Before measuring we wrote down a number: at 512, a 32,000-token prompt would take at least 1.5 times as long to reach its first token.

It took 1.27 times as long: 1.73 seconds against 1.36. Throughput with 16 requests at once moved by 2%. Two-thirds of the memory came back for a third of a second on a very long prompt.

That was one of seven predictions we committed before writing the measurement script. This post covers all seven: how fast one request runs on each runtime, what happens when requests arrive together, what prefix caching saves, and what a 4-bit file changes in llama.cpp. Along the way a llama.cpp default made 64 requests wait 144 seconds for their first token, which we had not planned to measure at all.

Setup. One A100 80GB, nothing else on the GPU. llama.cpp build 4da6337, llama-server -ngl 99 -fa on, f16 cache. vLLM 0.29.1rc1 as an OpenAI-compatible server with gpu_memory_utilization 0.85, CUDA graphs and compilation on (Part 1 measured memory in eager mode; here we kept vLLM's serving defaults). For the runtime comparison both sides load the same weights in bf16: vLLM from the Hugging Face checkpoint, llama.cpp from a bf16 GGUF we converted with the same build. Requests go to /v1/completions with streaming, temperature 0 and a fixed output length. Prompts are random English words, with a different start for every request so nothing is reused unless we say so. Every condition starts with a short warm-up request we throw away.

We measured three things. Time to first token (TTFT): from sending the request to the first generated token, which is mostly the time to read the prompt. Generation speed: tokens per second after the first one, for one request. Throughput: all output tokens from a batch of requests sent together, divided by the time the whole batch took.

1. One request: same generation speed, different prompt speed

Qwen3-8B, one request at a time, 256 output tokens, median of three:

Generation is the same on both: within 1% at both lengths. Generating one token means reading every weight once, and both runtimes do that at the same rate on this GPU. Both also slow down by the same amount at 32K, where every new token reads a longer cache.

Reading the prompt is where they differ. vLLM reached the first token in 0.38 times llama.cpp's time at 4K and 0.45 times at 32K. Part of that gap is not prompt processing, though. In llama.cpp only the first of the three requests was fast (0.45 s at 4K, 6.1 s at 32K); the second and third took 0.86 to 0.88 s and 9.1 to 10.3 s. Section 3 explains why. With the setting responsible turned off, llama.cpp took 0.44 s and 6.15 s on every repeat, and vLLM's TTFT was 0.75 and 0.67 times that. So vLLM reads long prompts faster on this GPU either way, but the gap at 4K shrinks from 0.38 to 0.75 depending on that one llama.cpp setting.

2. Many requests at once: where the runtimes split

Same model, N requests sent at the same moment, each with a 1,024-token prompt and 256 output tokens. llama.cpp ran with --parallel N and 2,048 tokens per slot. vLLM ran one server for every N.

Both runtimes batch. llama.cpp at 16 requests produced 5.5 times what it produced at one, so requests in different slots really are processed together. But vLLM's total kept climbing, to 18 times its single-request rate at 64. At 64 requests it produced 2.1 times llama.cpp's output with llama.cpp's prompt cache off, and 3.7 times with it on. Seen from one user, a request in a batch of 64 generated at 35 tok/s on vLLM and 16 on llama.cpp.

The 76.5-second TTFT in the llama.cpp row is not a typo, and it is not batching. The first of the two repeats finished all 64 requests in 21.5 seconds. In the second, every request waited 144 to 153 seconds for its first token. The throughput of 430 is the median of 761 and 100.

Memory behaves as Part 1 described. llama.cpp's GPU use, read from nvidia-smi under load, grew with the slots it was given: 15,337 MiB at one slot, 33,491 MiB at 64. vLLM sat at 69,585 MiB for every N, because it takes its 0.85 share at startup and fills it with cache whether one request comes or sixty-four.

3. llama.cpp's prompt cache in RAM: why the second batch waited

The server log shows what happened between the two batches of 64. Before a slot that still held the previous conversation could take a new request, the server wrote that slot's state into a prompt cache in host memory. Each entry was 179.875 MiB (one slot with 1,279 tokens of Qwen3-8B cache). The cache has a limit, --cache-ram, 8,192 MiB by default, so it was already full. For every new entry the server removed the oldest ones, logging each removal, and moved on to the next slot. It logged 1,000 removals, one every 0.13 seconds or so, before all 64 slots were running again.

The same step explains section 1. With one slot, each new request first saves the previous 4K or 32K conversation, and Qwen3-8B's cache is 144 KiB per token, so a 32K conversation is about 4.4 GiB. That is the extra 0.4 seconds at 4K and 3 to 4 seconds at 32K that only the first request avoided.

We checked the cause after seeing the numbers. We reran the same requests with --cache-ram 0, which turns the prompt cache off, and registered that run as a check outside the scored predictions. The repeats became identical: 0.44 s on all three 4K requests, 6.12 to 6.18 s at 32K, 21.2 and 21.5 seconds for the two batches of 64.

The cache exists for a reason. If a conversation comes back after its slot has been given to someone else, llama.cpp can restore it from RAM instead of reading the whole history again. We did not measure that benefit. What we can say is the cost, on this workload where no conversation ever returned: up to 4 seconds per long request with one slot, and a stall of over 144 seconds for every request once 64 slots had all been used. If your server mostly gets new, unrelated requests, try --cache-ram 0 or a smaller limit and compare. If conversations come back and slots are few, keep it and measure the restore against the save.

4. vLLM's memory reserve: what lowering it costs

Back to the opening question. Gemma 3 4B in bf16, three values of max_num_batched_tokens, with the per-request reserve from Part 1 alongside:

Lowering it slowed long prompts, by less than we predicted, and barely touched throughput with short prompts. For a sliding-window model like Gemma, 512 gives back most of the memory Part 1 measured for a third of a second on a 32K prompt. The 16-request batch shows why throughput hardly moved: its median first token came 0.4 to 0.7 seconds into a run of 3.3 to 3.6 seconds, so most of the time went to generating, which this setting does not change. For reference, Gemma 3 4B on llama.cpp read a 32,000-token prompt in 2.91 s (median, prompt cache on; the first request took 2.27 s).

One detail for deployment. Part 1's 2,682 MiB was measured through vLLM's Python LLM class, whose default on an A100 is 8,192. The vllm serve server defaults to 2,048 on the same GPU (16,384 and 8,192 on H100-class GPUs, per this build's source). So a default server on an A100 is already on the 1,242 MiB row. Check the value your deployment actually uses; in this build the log does not print it directly, but the compile range does (compile range (1, 2048)).

5. Prefix caching: reuse on a hybrid model

Part 1 found that prefix caching doubles the fixed state vLLM reserves for Qwen3.5-9B, a hybrid of attention and Gated DeltaNet layers. The question left was whether it pays that back. We sent eight requests one after another, each with the same 8,000-token start and a different 64-token end, 16 output tokens each.

With caching on, requests after the first reached their first token in 0.13 times the time it took with caching off. The shared 8,000 tokens were read once and reused, so the linear-attention state was restored rather than recomputed. vLLM logged the state cache in its "align" mode, the mode that costs the second block per group in Part 1. For workloads with long shared starts (a fixed system prompt, a document several questions refer to), that memory buys a reread of about 0.6 seconds per request here.

llama.cpp also reused most of the start. Its single slot still held the previous request, and the log shows it reading 516 tokens of each 8,064-token prompt again, not all of it. That is more than the 64-token tail; we did not confirm why it went back further. It ran a 4-bit file, so compare the numbers in its row with each other, not with vLLM's.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

6. A 4-bit file in llama.cpp: faster generation, slower prompts

Most llama.cpp deployments do not run bf16; they run a quantized GGUF. Qwen3-8B Q4_K_M against bf16 on the same server settings:

Generation was 1.5 times as fast at 4K. We had predicted 1.8 times, reasoning that a file a third the size should be read nearly three times as fast per token. Prompts went the other way. At the median, Q4_K_M took 1.6 times as long to reach the first token at 4K and 1.4 times at 32K, and it was slower in every one of the six pairs of repeats. Reading a prompt is matrix multiplication over many tokens at once, and there 4-bit weights have to be unpacked rather than simply read. We did not profile the kernels, so take that as the likely reason, not a measured one. In practice, a 4-bit file mainly saves memory and speeds up generation; it does not make long prompts faster on a GPU like this one.

Predictions, scored

We committed these, with their thresholds, before writing the measurement script. Scoring uses the registered conditions (llama.cpp defaults, prompt cache on).

Q2 and Q4 pass by a margin that depends on llama.cpp's prompt cache. With --cache-ram 0 (our check after the fact, not scored) Q2 would have been 0.753 at 4K and 0.67 at 32K, just failing its 0.75 threshold at 4K, and Q4 would have been 2.06, just passing.

What this means for choosing and configuring

These are one GPU's numbers with two models and fixed workloads, so read them as a direction to measure your own case, not as a rule.

• One user, or a few. Generation speed is the same on both runtimes at the same precision. The differences are elsewhere: llama.cpp runs a 4-bit file (1.5 times faster generation here, less memory) and holds only the memory its slots need, while vLLM reads long prompts faster.
• Many users at once. This is where the runtimes split. At 64 concurrent requests vLLM produced 2.1 to 3.7 times as much and each request kept twice the generation speed. With llama.cpp, set --parallel to your real concurrency and remember that -c is divided across the slots (Part 1).
• llama.cpp, many unrelated requests. Test --cache-ram 0 or a smaller limit. On our workload the default made repeat requests slower, up to a 144-second stall at 64 slots.
• vLLM, sliding-window models. Lowering max_num_batched_tokens recovers most of the memory for a small cost on long prompts (Gemma at 512: 882 MiB instead of 2,682 for one 32K request, 0.37 s slower on a 32K prompt). Know which default your entry point uses.
• vLLM, long shared prompt starts. Keep prefix caching on, including on hybrid models. It costs a second state block per group and saved most of the reread here.

Limits

• One A100 80GB, one build of each runtime, CUDA graphs on in vLLM. Batching, kernels and the prompt cache change between versions.
• Three models (Qwen3-8B, Gemma 3 4B, Qwen3.5-9B), each in one or two of the measurements. Repeats were three for one request and two for batches.
• Random-word prompts, fixed output lengths, requests sent all at once or one after another. Real traffic arrives spread out, with varied lengths and returning conversations; we measured none of that, and none of the benefit of llama.cpp's prompt cache.
• No answer quality. Temperature 0 and fixed lengths were for timing only.
• The --cache-ram 0 runs were added after we saw the results and are not part of the scored predictions.
• The reason given for slower prompts with Q4_K_M is not profiled.