Same Model, Five Ways to Build the Input: Only Retrieval Moved It Significantly

Same Model, Five Ways to Build the Input: Only Retrieval Moved It Significantly

An audit of SWE-bench leaderboards reports that within-model scaffold ranges reach 29.8 percentage points, against an 8.8-point spread across the top thirty entries. In that paper a scaffold is the whole system around the model — agent loop, tools, retrieval, control policy — and the authors state plainly that their observational design does not identify causal scaffold effects.

This post asks a much narrower question. Hold the model and the data fixed, change only how the classification input is assembled, and see how far the number moves. The task is 77-way intent classification, not coding agents.

First, the same configuration twice more

Before comparing anything, I ran the identical configuration three times: same prompt, same 154 messages, reasoning effort low. It scored 125, 127 and 128 correct. An earlier run of the same prompt, from the Jev post three days before, scored 129. Totals move without changing anything.

Minimum and maximum over three or four runs is a weak basis for comparison, so I follow the method the cited paper uses: compare the same items pairwise and apply an exact McNemar test. The baseline is fixed as the 127-correct run.

What each scaffold scored

GPT-5.6 Terra, reasoning effort low, output constrained to one of the 77 labels, 154 messages (two per intent) from BANKING77's test set. Examples come only from the training split. The table holds five ways of building the input plus two controls that change only the reasoning effort.

By totals alone the spread runs from 79.9% to 94.8%. Compared item by item against the baseline it looks different:

Shuffling the label list, grouping it by topic and changing reasoning effort were all non-significant against the baseline under an exact McNemar test. Grouping broke more than it fixed, five against one, and that difference does not reach significance either.

Retrieval is the exception, and it is not a small one.

The retrieval result, compared message by message

The retrieval scaffold embeds the incoming message with a pretrained MiniLM model, pulls the five nearest messages from the 10,003 training examples, and pastes them above the question with their labels. The model still answers with one label.

Against the baseline run on the same 154 messages: both correct on 127, retrieval alone correct on 19, baseline alone correct on 0, both wrong on 8. It fixed nineteen items and broke nothing; the exact two-sided McNemar p is about 3.8e-6. That is not items trading places, it is one run covering the other.

Two things are worth separating here. Retrieval did not make the prompt better written. It made labelled data reach the model at inference time. The 12-point jump is the value of the training set, not of prompt wording — which is the same conclusion as the classifier that learned from those labels directly and reached 90.3% on this sample. Retrieval reaches 94.8% from the same examples with no task-specific training or fine-tuning, though the embedding model it uses is itself pretrained.

The cheapest run is not the one with the smallest prompt

The 77-example scaffold sends 2,406 tokens per request, 2.8 times the baseline, and costs less than a third as much: $0.73 per thousand messages against $1.97. The prefix is identical on every request and long enough to be cached, so 99% of its input tokens were served as cached input at a tenth of the price. The baseline prompt, at 873 tokens, never cached at all.

So a longer prompt can be the cheaper one, if it is the same on every request. The retrieval scaffold cannot have that, since its examples change per message, which is why it is the most expensive row despite sending fewer tokens than the 77-example one.

What this does and does not say about the 29.8 points

The spread across my five variants is 15 points, from 79.9% to 94.8%. That is about half the paper's within-model figure, and far larger than the one-item gap between two frontier models I measured on this task earlier.

But the shape matters. Four of the five showed no significant paired difference from the baseline. The entire spread came from one change, and that change was not phrasing — it was giving the model labelled examples. On this task, rewording moved nothing I can measure, and data reaching the prompt moved everything.

This does not confirm the paper's finding. The task is different and its notion of a scaffold is far broader than anything I varied. What carries over is the method: do not read total-score gaps as an ordering, compare the same items pairwise.

What this does not show

One dataset, one model, one run per variant except the baseline, and 154 messages. Each p-value tests the item-level difference between those two runs only; it does not account for the model's run-to-run variation. They are computed per comparison with no correction for multiple testing; the retrieval result survives correction, the others are not significant to begin with. The retrieval numbers exclude the embedding and lookup time, which ran on CPU outside the API call. Cached-input pricing depends on traffic patterns, so the cost column describes this run rather than a production bill. And the labels in BANKING77 are fine-grained and often near-synonyms, which is exactly the case where nearby examples help most.

Setup: BANKING77 test set (PolyAI, CC-BY-4.0), 154 messages, two per intent, seed 20260918. Model gpt-5.6-terra through the Responses API with the 77 labels as an enum, reasoning effort low unless stated. Retrieval uses pretrained all-MiniLM-L6-v2 embeddings over the 10,003 training messages, cosine similarity, five neighbours. Prices from OpenAI's model page: $2 input, $0.2 cached input, $12 output per million tokens. Total API spend for this post: $2.62. Measured on 2026-09-21.