Why ChatGPT Gives You a Different Answer Every Time -- One Knob, Measured

Why ChatGPT Gives You a Different Answer Every Time -- One Knob, Measured

We measure, on a local model and on GPT-4o-mini, exactly how much answers change with sampling temperature. Temperature doesn't create diversity -- it amplifies what's already there.

You ask the same question twice and get two different answers. Yesterday it gave you a table; today it writes paragraphs. This isn't a bug. It's by design. In this post we reproduce that design on a local open model (Gemma 2 2B), turn the one knob responsible -- sampling temperature -- and measure exactly how much the answers change.

This is Part 1 of the "How AI Actually Works" series. The rule of the series is simple: take one strange thing you actually experience with AI, and answer it with an experiment we ran ourselves.

1. The model doesn't choose an answer -- it draws one

The only thing a language model like ChatGPT does is this: look at the text so far and compute a probability distribution over the next token (word piece). After "The sky is blue because", it might assign 62% to "light", 21% to "the atmosphere", 9% to "scattering"... and then it rolls a die to pick one.

Temperature is the knob that controls how extreme that die is.

• Temperature 0: no die. Always pick the single most likely token. Same question → same answer, every time.
• Temperature 1: draw according to the computed probabilities as-is. A 62% token gets picked about 62 times out of 100.
• Above 1: deliberately flatten the differences so weaker candidates get a shot. More creative -- and more likely to go off the rails.

That much is in every textbook. Now let's measure what it actually does.

2. Experiment setup

The three prompt types are the whole point of this experiment: temperature behaves differently on questions with one right answer versus questions with many.

3. Results: temperature is an amplifier, not a source

!Mean pairwise similarity between responses, by temperature

*Each cell: mean pairwise cosine similarity between responses (distinct-answer count in parentheses). Temperature 0 is deterministic by definition, so it's omitted.*

Three things to read out of this.

First, temperature cannot create diversity that isn't there. "What is the capital of Australia?" comes back "Canberra" 200 times out of 200 all the way up to temperature 1.0. It first splits at 1.3 -- and even then barely (similarity 0.979). When the model's distribution is concentrated on one token, no amount of die-rolling changes the answer. The diversity lives in the distribution; temperature only decides how much of it you see. (An earlier 30-sample run showed a single deviation at 1.0; at 200 samples it vanished -- with n=30 you cannot tell a ≤3% event from noise.)

Second, the moment the answer becomes a sentence, it splinters even at low temperature. The explanatory prompt produces 18 distinct sentences out of 30 at temperature 0.3. The reason is compounding, and this time we measured it: teacher-forcing the greedy answer (27 tokens) and reading off the probability of the top token at each position gives an average of 93.3% at temperature 0.3 -- which sounds high, until you multiply 27 of them: the chance of reproducing the exact sentence is 10.8%. At temperature 1.0 it's 80.1% per token and 0.08% for the whole sentence. The real mechanism behind "it answers differently every time" is not temperature -- it's answer length.

Third, the gap between the orange and green lines in the chart above is the most important number in this post. The explanatory answers splinter into 30 different sentences, yet their similarity stays at 0.89–0.93 -- read them and they all say Rayleigh scattering, just worded differently. The creative answers drop to 0.57–0.68 -- genuinely different stories. Most of "a different answer every time" is a different sentence, not a different fact, and that difference is now quantified. When the facts themselves diverge, that's a separate problem (hallucination -- coming up in this series).

One honest footnote: the explanatory line ticks slightly upward between 0.7 and 1.0 (0.907 → 0.911). That's within n=30 sampling error and doesn't affect the trend.

4. The same experiment on the ChatGPT family -- GPT-4o-mini, measured

The obvious objection: "that's a 2B local model." So we repeated the experiment through the OpenAI API (gpt-4o-mini) -- factual and explanatory prompts, five temperatures, 30 calls each.

Same pattern as Gemma. The factual answer is identical 30/30 up to temperature 0.7; the explanatory one splinters into 16 sentences at 0.3 while similarity stays at 0.94 (same content, different wording). What we measured is not a small-model quirk -- it's a general property of autoregressive sampling.

And note the bold cell: at temperature 0, the explanatory prompt still produced 4 distinct answers. Section 6 warns that temperature 0 doesn't guarantee perfect reproducibility -- that server-side nondeterminism just showed up in the data.

5. So why does ChatGPT vary?

ChatGPT's default temperature is not 0 (the API default is 1.0; the web app's internal value is unpublished, but its outputs clearly aren't greedy). Why not just set it to 0?

• Temperature-0 answers are safe but flat. Summaries, translations, and creative writing get noticeably monotonous.
• It repeats the same mistakes identically every time. Randomness is also a retry mechanism.
• A chatbot that answers the same question the same way every time reads as "robotic" to users.

The variation you experience is randomness deliberately left in, for quality.

6. What to do with this in practice

• Work that needs reproducibility (data extraction, classification, grading): if you're on the API, temperature 0 or ≤0.2. The ChatGPT web app has no temperature control -- your next-best lever is strict output formatting.
• Ideas, drafts, style variations: raising temperature (API) or simply asking multiple times and picking the best is a legitimate strategy. The randomness is a feature.
• One caveat: temperature 0 still doesn't guarantee perfect reproducibility (server-side nondeterminism exists). It means "almost always the same", not "always".

7. What this experiment does not tell you

• This still isn't the ChatGPT web app itself. The main experiment is Gemma 2 2B, validated on the GPT-4o-mini API. The web app's internal temperature and filter settings are unpublished, so your mileage may differ in degree.
• Embedding similarity approximates semantic sameness. BGE-M3 isn't a perfect judge of "same content" -- which is why the raw samples are included.
• Three English prompts. Korean, long prompts, and code may have differently shaped distributions.

8. Takeaways

1. The model doesn't choose -- it draws from a probability distribution. Temperature sets how extreme the die is.
2. Temperature is an amplifier. One-right-answer questions barely split even when it's high; many-right-answer questions split even when it's low.
3. The same pattern reproduces on GPT-4o-mini -- which even at temperature 0 produced 4 distinct answers. Perfect reproducibility was never on the table.
4. "Different answer every time" is not a bug -- it's a quality trade-off. If you need reproducibility, lower the temperature or lock the format.

*This post answered one question. For the whole system -- sampling, hallucination, context, detection -- there's a video course (Korean available now, English edition in review): see courses →*

References

• Holtzman et al., The Curious Case of Neural Text Degeneration (ICLR 2020) -- sampling strategies and text quality
• Experiment code, seed, and raw results: download the companion notebook at the bottom of this post (seed 42 -- reruns reproduce these numbers)