Ask a chatbot the same question twice and you will often get two different answers. That is not a glitch. A language model does not look up a stored reply; it builds its answer one token at a time by choosing from a probability distribution, and the settings that govern that choice, such as temperature, top-k and top-p, decide how adventurous the model is. Understanding them helps you get more reliable results for factual work and more variety for creative work.
From scores to probabilities
At each step the model produces a score, called a logit, for every token in its vocabulary. These scores are converted into probabilities with a function called softmax, so that they are positive and add up to one. The model then picks the next token and repeats. The simplest strategy, greedy decoding, always takes the highest-probability token. It is deterministic but has a known weakness. In their widely cited 2019 paper “The Curious Case of Neural Text Degeneration” (published at ICLR 2020), Ari Holtzman and colleagues observed that although maximizing likelihood trains good models, using likelihood as the decoding objective “leads to text that is bland and strangely repetitive.” Their finding was that the decoding strategy alone can dramatically affect text quality, even from exactly the same model.
So most systems sample instead: they draw the next token at random, weighted by its probability. A token with 60% probability gets picked about 60% of the time. That randomness is why answers differ between runs.
Temperature: sharpening or flattening
Temperature is a number that divides the logits before softmax. A temperature of 1 leaves the distribution as the model produced it. Lower values sharpen it, making the top choice even more dominant; higher values flatten it, giving unlikely tokens more of a chance.
Here is a small worked example with three candidate next words and logits of 2.0, 1.0 and 0.1:
| Temperature | Word A | Word B | Word C |
|---|---|---|---|
| 0.5 | 86.4% | 11.7% | 1.9% |
| 1.0 | 65.9% | 24.2% | 9.9% |
| 2.0 | 50.2% | 30.4% | 19.4% |
(Values are rounded and calculated by dividing the logits by the temperature, then applying softmax.) At 0.5, word A wins almost nine times in ten; at 2.0, the least likely word is picked about one time in five. Near zero, behavior approaches greedy decoding. This is why low temperature suits tasks with a single right answer, such as extracting data or writing code, and higher temperature suits brainstorming, where variety is the point.
Top-k and top-p: cutting off the tail
Temperature reshapes the whole distribution, but a long tail of very unlikely tokens still exists, and sampling from it can produce nonsense. Two truncation methods address this.
Top-k keeps only the k most probable tokens and renormalizes. If k is 40, the model chooses among the 40 best candidates. The weakness is that a fixed k does not adapt: sometimes the model is nearly certain and 40 candidates is too many; other times it is uncertain and 40 is too few.
Top-p, also called nucleus sampling, is the method proposed by Holtzman and colleagues. It keeps the smallest set of tokens whose probabilities add up to p, for example 0.9. When the model is confident, that set might be one or two tokens; when it is unsure, the set grows. In the authors’ words, sampling “from the dynamic nucleus of the probability distribution” allows diversity while truncating the unreliable tail, and the resulting text better resembles human text without sacrificing fluency and coherence. Many APIs expose top-p as a setting alongside temperature.
Practical settings
- Factual answers, extraction, code: use low temperature, often between 0 and 0.3, so the model sticks to the most probable tokens.
- Writing, brainstorming, naming: a moderate temperature around 0.7 to 1 gives variety while staying coherent.
- Changing both temperature and top-p: most API documentation suggests adjusting one at a time, because the effects overlap.
- Check provider rules: some newer models, particularly ones with built-in reasoning, restrict or ignore these settings, so read the documentation for your model.
Why “temperature 0” is still not perfectly repeatable
Even at the lowest setting, you can sometimes see small differences between runs. Floating-point arithmetic on GPUs can produce tiny numerical differences depending on batching and hardware, and those can flip a close call between two tokens. If you need guaranteed reproducibility, pin the model version, log inputs and outputs, and do not rely on determinism alone. For outputs that must follow a strict format, use structured outputs rather than hoping the sampler cooperates.
Why this matters for users
Sampling explains several everyday experiences. It explains why regenerating an answer can fix a wrong one, and why “creative” modes feel livelier. It also explains one cause of hallucination: when a distribution is flat, plausible-sounding but wrong tokens have a real chance of being picked. Settings cannot fix a model’s knowledge gaps, though. A low temperature makes the model repeat its best guess consistently; it does not make that guess correct. For background on the machinery that produces the probabilities, see how transformers work, and for the units that appear in model announcements, parameters, tokens and context windows.
What to watch next
- Provider changes that lock or hide sampling controls on reasoning models.
- New decoding methods, including ones that adapt truncation to model uncertainty.
- Better tools for reproducible generation in production systems.
- Research connecting sampling choices to hallucination rates.
Key takeaways
- Models choose each next token from a probability distribution, so answers vary between runs.
- Temperature sharpens (low) or flattens (high) the distribution.
- Top-k and top-p cut off unlikely tokens; top-p adapts to the model’s confidence.
- Use low temperature for factual or code tasks and higher for creative ones.
- Low temperature gives consistency, not correctness.
Frequently asked questions
What temperature should I use?
Start low (around 0 to 0.3) for factual or code work and around 0.7 to 1 for creative writing, then adjust based on results.
What is the difference between top-k and top-p?
Top-k keeps a fixed number of top tokens; top-p keeps the smallest set whose probabilities add to a chosen total, so the number varies with the model’s certainty.
Does temperature 0 make a model always right?
No. It makes the model pick its most likely token each time, which is consistent but can be consistently wrong.