Set temperature to 0—will the model be more accurate? First look at how it picks the next word

Break apart the process by which an LLM generates text, and it does not directly choose a word. At each step, the model first gives every token in the vocabulary a raw score, then turns those scores into probabilities, and finally samples one token from the probability distribution. Temperature, top-k, and top-p all act at this sampling stage; they do not change the model weights.

Akshay drew this pipeline as a diagram. In the video you can directly see how the sampler handles candidate tokens:

The division of labor among the three parameters can be seen as follows.

Temperature rescales the raw scores. A low value makes the distribution sharper, so a few tokens take up most of the probability. A high value makes the distribution flatter, so less popular tokens also have a chance to surface.

Top-k keeps only the k tokens with the highest probabilities. If k is set to 50, all tokens ranked after 50 are discarded. Regardless of whether the current distribution is sharp or flat, the number of candidates is fixed.

Top-p keeps the smallest set of tokens whose cumulative probability reaches at least p. If p is set to 0.9, and the next token is very clear, only a few candidates may be left. If the context is open-ended, many more candidates will remain.

The final step is sample: the sampler picks one from the remaining tokens.

The key difference is that top-k always keeps a fixed number of tokens, while top-p adjusts how large the candidate set is based on the model's confidence. Some commenters noted that with top-k fixed at keeping 3 candidates, and top-p truncated when cumulative probability reaches 0.9, examples like cat and mat make the difference easier to see.

Low temperature is often treated as a “smarter” knob, but this is easy to misread. Akshay puts it plainly: low temperature does not make the model more correct; it only reduces randomness. If the context is wrong, the model will still confidently give wrong answers. A commenter added a production scenario: a grading tool at temperature 0.0 may give the same wrong score every time, and a teacher checking twice may instead read this consistency as accuracy. Repeated runs only show randomness, not correctness.

If you want to start tuning from defaults, you can use this set of starting points:

• Classification and extraction: 0.0 to 0.2
• Code generation: 0.0 to 0.3
• General chat: 0.5 to 0.8
• Brainstorming: 0.7 to 1.2

Some commenters said that for code tasks, once temperature exceeds 0.3, the model starts making up method names. This rule of thumb may not apply to every model, but the direction is worth remembering. Code and extraction want convergence, brainstorming wants divergence, and temperature should not be a one-size-fits-all value.

The tuning order also matters. Akshay recommends tuning temperature first, then using top-p to prune the long tail of low-probability tokens. During evaluation, change only one setting at a time; do not turn all three at once. A commenter mentioned a common production pitfall: raising both temperature and top-p high at the same time makes the two knobs fight each other. For exploratory tasks, choose one main knob and leave the other near its default value unless you have measured the task benefit.

There is also a switch that is easy to overlook. A commenter reminded that in transformers, if do_sample=false, temperature will not take effect. Before comparing 0.2 and 0.8, first confirm that sampling is actually enabled.

These parameters only affect the sampling stage. Looking before and after, LLM inference also includes tokenization, embedding, attention, the prefill and decode stages, KV cache, and quantization. In his long article, Akshay breaks down this pipeline in detail: prefill is compute-bound, decode is memory-bound, KV cache consumes GPU memory as generation length grows, and quantization trades lower bit width for speed and memory. People have also put together a production optimization map. Avi Chawla shared a fairly comprehensive overview:

Back to the most practical question: when the model behaves incorrectly, first distinguish whether it “starts slowly” or “streams slowly,” then distinguish whether sampling is too random or the context itself is insufficient. Setting temperature to 0 will not fill in missing information; it will only make the model follow the existing probabilities more consistently.

Akshay's complete inference breakdown is here: http://x.com/i/article/2050553855281451008