● A free public tool  ·  Built by Poptani Labs  ·  No sign-up  ·  Your data never leaves your browser
AI, Decoded · 5-minute read

Why the same prompt gets a different answer every time

An artificial intelligence and robotics exhibition at the Heinz Nixdorf MuseumsForum
Image: 'Artificial Intelligence (AI) and Robotics exhibition at the Heinz Nixdorf MuseumsForum.jpg' by Sergei Magel/HNF, CC BY-SA 4.0, via Wikimedia Commons

Ask a chatbot the same question twice and you will get two different answers. Sometimes the difference is small — a different turn of phrase, a different example. Sometimes it is large enough to feel like you are talking to a different machine. It can feel like a bug, or like the model is being moody. It is neither. The variation is built into the way these models write, and it is controlled by a setting most users never see: temperature. This is a research-based explainer — no hands-on testing, just how the machinery works and what the evidence shows.

The model doesn't look up answers — it rolls dice

A large language model does not store answers and retrieve them. When you send a prompt, it predicts the next word — or more precisely, the next token, a chunk of a word — then feeds that prediction back in and predicts the next one, thousands of times, until the reply is finished. At each step it produces a probability for every possible next token: the most likely word might have a 40% chance, the next 12%, and so on down a long tail of unlikely options. The model then samples from that distribution — it effectively rolls a weighted die. The same prompt can therefore take a different path through the probabilities on every run, and the replies diverge. As explained in our earlier look at how next-token prediction works, the model is a very sophisticated autocomplete, not a search engine — and autocomplete with a die in it will never say the same thing twice in exactly the same way.

Temperature: the dial that controls the dice

Temperature is the number that decides how heavily the dice are weighted. Before sampling, the model's raw scores (called logits) are divided by the temperature, then turned into probabilities. A low temperature — say 0.2 — sharpens the distribution, so the most likely token dominates and the output becomes predictable, even repetitive. A high temperature — say 1.5 — flattens it, so unlikely tokens get picked more often and the output gets creative, strange, or wrong. A temperature of 1.0 leaves the distribution as the model learned it. In OpenAI's API the range is 0 to 2 with a default of 1.0. A related setting, top-p or nucleus sampling, works differently: it keeps only the smallest set of tokens whose probabilities add up to p (usually 0.9) and samples from within that set — confident moments keep two or three candidates, uncertain moments keep hundreds. The technique comes from a 2019 paper by Holtzman and colleagues, "The Curious Case of Neural Text Degeneration", which proposed it as a fix for the repetitive loops early models fell into. OpenAI advises changing temperature or top-p, not both at once. And the newest reasoning models ignore both settings entirely, telling developers to steer them with reasoning effort and prompting instead — which is one reason the same question can feel different across model generations as well as across runs.

Even zero isn't perfectly predictable

Here is the part that surprises people: even with the temperature turned all the way down to zero — which should, in theory, always pick the single most likely token — the output is still not guaranteed to be identical. Researchers at King's College London, UCL and Bristol tested this properly. They sent the same 829 coding problems to ChatGPT five times each at the default settings, and on the CodeContests set, 75.76% of the problems produced no two identical outputs across the five runs. Then they turned the temperature to zero and ran it all again — and still, 43.64% of the CodeContests tasks produced no identical outputs across five runs. The study, published in the ACM Transactions on Software Engineering and Methodology in January 2025, is the finding that settles the question: even "deterministic" mode isn't. The causes are mundane rather than mysterious — floating-point arithmetic that gives slightly different results depending on how the work is split across GPU cores, requests batched together dynamically, and sparse "mixture of experts" routing that sends a prompt down a different internal path each time. Anthropic's own documentation says plainly that "even with temperature of 0.0, the results will not be fully deterministic", and OpenAI describes its seed parameter as "best effort" with no determinism guarantee. The dice, in other words, are loaded — but they are still dice.

Why your chatbot feels moody

None of this is visible in the apps most people use. ChatGPT, Gemini and Claude on the web do not let you set a temperature; they run above zero by default, because a chatbot that always gave the same answer would feel robotic and would never surprise you with a better phrasing. So the variation you notice is not the model changing its mind, and it is not you imagining things. It is the sampling step doing exactly what it was designed to do — and it is why re-asking a question is a legitimate strategy: a second roll of the dice sometimes lands on a clearer, shorter, or more accurate answer. The same mechanism is behind the model's tendency to invent facts, as we covered in why AI sometimes makes things up — confidence and correctness are different things, and the sampler does not know which is which.

What This Means

Variation is not a flaw in the machine; it is the machine. A chatbot's answer is one sample from a distribution of plausible continuations, shaped by temperature and top-p, and even the most careful settings cannot make it perfectly repeatable. Treat a single answer as a draft, not a verdict: ask again for a second opinion from the same model, and check anything that matters against a source that doesn't roll dice.

Related on Poptani Labs: ChatGPT is autocomplete on steroids: how next-token prediction actually works, Why AI sometimes makes things up — and why it sounds so sure, What are AI agents, really? The difference between a chatbot and an agent. More in this category: AI, Decoded.

Sources: Why AI models give different answers every time (AnswerBurst), LLM temperature sampling, explained, Temperature, top-p and top-k, explained, An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (arXiv), Non-determinism in AI LLM output

← All posts