DiffusionGemma on vLLM's Example Server: One Sentence of Context Took It From 54.5% to 31.8%

DiffusionGemma on vLLM's Example Server: One Sentence of Context Took It From 54.5% to 31.8%

A diffusion language model fills a whole canvas at once instead of writing left to right. A vLLM pull request merged on September 22 uses that: seed the canvas with an answer template, leave the answer slot noisy, run one denoise step, and read the distribution over that slot. You get a decision with probabilities and no token-by-token generation. That is the shape TypeSafe's Jev sells as a "System One" model, here on open weights.

I ran the PR's example server on a task I had already measured: BANKING77, 77 intents, the same 154 messages. I also wrote the same read on transformers, so the two can be compared message by message.

Fixed at one noise draw, the example server lands 8 points below a grammar-constrained 8B model, about 14 points below my version of the same read, and 36 points below a logistic regression on a CPU. With its default draws the gaps are 11, 17 and 39 points. It is also the fastest way I have run a 26B model on this task.

The server takes at most 26 options per question

The example server, structured_server.py, labels a question's options A to Z itself and rejects anything longer. I sent all 77 intents as one question to check. Every request came back 422: question 'intent': at most 26 alternatives. The PR description names the underlying rule: each answer must be a single token so the canvas does not shift, and a client can map longer options to letters.

So 77 intents have to be split, and the server supports that directly. A question can carry ask_if, which makes it run only when an earlier answer matches. I sorted the intent names in case-sensitive ASCII order, which puts Refund_not_showing_up first, and cut the list into groups of up to nine; the last group has five. I sent one group question plus nine intent questions, each asked only if its group was picked. Contiguous names keep related intents mostly together, though not entirely: eight of the ten card_* intents share a group and the other two fall into the next one.

By default the server decides how many noise draws to average. If the first read's entropy is above 0.1 it takes four draws per stage, and 130 of the 154 messages took eight reads in total. Fixing the server at one draw (samples: 1) changed 23 of the 154 answers, moved accuracy from 79 to 84 correct (p = 0.27, not significant), and cut the median from 325 ms to 119 ms.

Getting it to run on an A100

The PR targets vLLM 0.29, whose kernels link against CUDA 13. That needs a 580-series driver; this machine had 535, so I upgraded it and moved torch to the cu130 build.

I first ran the branch at its pre-merge commit 407326b. There, FlashInfer was auto-selected and crashed while capturing CUDA graphs, and FlashAttention rejected the model's 512-wide attention heads, so I had to pass --attention-backend TRITON_ATTN. Two other people hit the same crash on an L40S and a GB10 and reported it in the PR, and it was fixed before the merge. At the merged commit the server picks Triton on its own and starts on an A100 with no backend flag.

The numbers in this post come from the merged commit. At 407326b the same two requests scored 88 and 86 instead of 84 and 79. One draw at each commit differed on 25 answers, but neither beat the other significantly (9 against 5, p = 0.42).

Same model, same groups, fourteen points apart

My transformers version reads the same slot the same way, with one denoise step. With the same nine groups it picked the right group for 80.5% of messages and finished at 105 of 154. The example server with one draw picked the right group 68.8% of the time and finished at 84.

The two agree on 99 of 154 predictions. Of the messages only one of them got right, mine got 29 and the server got 8. An exact McNemar test gives p = 0.0008 for that split, and p = 0.0001 against the default draws (34 against 8). Run-to-run noise does not explain this: the server with one draw returned the same 154 answers when I ran it twice.

What differs is the prompt around the read. The example server writes a generic system message ("Answer a fixed set of questions about the state the user provides") and lists options as A: card_arrival. Mine opens with "You are a banking customer-support intent classifier" and gives each intent a one-character symbol. The server also restates the first answer before the second read. I did not isolate which of these accounts for the gap.

Update (2026-09-23): A follow-up measurement ruled out the system message. With the example server's generic prompt, openjev read all 77 options at once and scored 103 of 154. The gap sits in the example server's two-stage implementation. Details.

One sentence of context cost the server 20 points or more

The obvious fix is to tell the server what the task is. Its schema accepts a top-level instructions field, which the server adds as one line to the system message and uses nowhere else; I checked the code. I tried two sentences, one draw each:

Both made it worse, by 23 and 26 points. The second sentence tells the model to pick an intent while the first question's options are groups, so a conflict there is at least plausible. The first sentence only describes the input, and it still cost 35 messages.

I cannot tell you from this which prompt is right. It does show how far a one-step read moves on wording: one line mattered far more than any sampling setting in this post. A tester in the PR thread, vineethsai7, found something similar on their own data. Across the four conditions they published, their F1 ranged from 0.51 to 0.90, and they wrote that "schema choice dominates", with a gap "much larger than any context change" (for them, context meant adding prior conversation turns). If you use this read, measure your own prompt on your own data. A number from someone else's prompt tells you little.

My version's knobs, and the CUDA version

The two rows of the first table that compare the implementations were both measured on the new driver, so the gap between them is not a CUDA difference. The question that remains is whether the knob table below, which I measured on the old driver with torch cu126, still holds. I reran its default condition on cu130: all 154 predictions were identical.

Three more things barely mattered, measured on the new driver. Dropping the empty thought block the PR puts at the start of the canvas (<|channel>thought\n<channel|>) gave 102 instead of 105 (p = 0.45). Reassigning the 77 symbols at random three times gave 103, 104 and 102. And one read against two stages was 105 against 105, each winning ten messages the other lost.

What the PR measured and what I measured

The PR reports 8.7 requests per second at 0.12 s for a single request on a DGX Spark, and 54 requests per second at 0.58 s with 32-way concurrency. That run used the NVFP4 checkpoint, a 32-token canvas, and three decisions per request. Mine used bf16 weights, a 64-token canvas, a 1,600-token prompt, and two stages per request, on an A100. It came to a 119 ms median and 8.49 requests per second, one at a time. The figures look alike, but with that many differences I am not calling it a reproduction.

The PR is not without accuracy numbers. Its description has small hand-built suites: programming language 10/10, human language 9/10, unit comparison 10/12, plus vision tasks such as colors and counting. Its commit messages measure consistency. When two dependent choices sit in one canvas, they agreed on 18 of 24 noise draws at eight unpinned steps and on 24 of 24 when pinned. Answering them in sequence took agreement from 17 of 24 to 24 of 24. The thread adds more. pst2154 built a 72-case diagnostic and reports 71 of 72 (98.6%) against 55 for NanoJev, and calls it "not a broad benchmark". vineethsai7 reports F1 over 12,152 requests on their own cases, and Gibcity a 98.6% language-ID smoke test.

What I did not find, in the description or in any comment, is accuracy on a public dataset scored against human labels. Most of those suites ask a handful of options; BANKING77 asks 77 through the 26-option limit. That is the gap this post fills, for one task.

Jev's own evaluation reports 67.8% across its four workflows, almost exactly my version's 68.2%. The two measure different tasks against different references: theirs against two frontier models' averaged answers, mine against human labels. Side by side they are a coincidence, not a comparison.

Where this shape fits

The probabilities are the interesting part. My version averaged 0.95 on its chosen answer, and both it and the example server at one draw returned identical predictions on a rerun. So a threshold can route confident cases one way and send the rest to a slower path. I did not measure whether the probabilities are calibrated. With the default four draws, vineethsai7 measured answers flipping on 19 of 804 identical requests (2.36%); I did not repeat that setting.

Seventy-seven fine-grained options is also not where the PR aims. Its examples ask binary or few-way questions: is this urgent, does it need a human, which of three queues. Those fit in one question under the 26-option limit, and that is the shape to test first.

What this does not show

One dataset, 154 messages, one model, and a handful of prompts. I changed the example server only through its request schema, and the groups are my choice. I did not isolate which part of the prompt causes the fourteen-point gap. The PR's DGX Spark numbers I could not check on this hardware. The model takes 48.1 GiB of an A100 at bf16 in transformers; the vLLM server reserves most of the card.

Setup: google/diffusiongemma-26B-A4B-it (25.8B parameters, bf16). Example server: vLLM PR #57250 merge commit 1b3b88e, examples/features/structured_diffusion/structured_server.py, canvas 64, Triton attention picked automatically, driver 580.178.04, torch 2.13.0+cu130, one A100 80GB PCIe, requests sent one at a time, prompt 1,600 tokens at the median. The Python code is at the merge commit; the compiled kernels are from the pre-merge build (0.29.1rc1.dev453+g407326b73), and the kernel sources differ between the two only in two files unrelated to this model (an all-reduce kernel and a Hadamard-transform fix on a quantization path). My version: transformers 5.17, canvas 256, one denoise step (template pinning applies only past one step), argmax over single-token symbols. The first table's rows for my version ran on the same driver and torch as the server; the knob table ran on driver 535 with torch 2.13.0+cu126. BANKING77 test set (PolyAI, CC-BY-4.0), 154 messages, two per intent, seed 20260918. Exact two-sided McNemar tests on paired predictions. Measured 2026-09-22 and 2026-09-23.