Temperature=0 Still Not Deterministic? Same Prompt Run 1000 Times, 80 Different Answers

Temperature 0 is often treated as the determinism switch for LLMs. At temperature 0, the inference engine switches to greedy decoding and picks the token with the largest logit at each step. The same prompt should, in theory, produce the same output every time.

In practice, it does not. Avi Chawla recently broke down this problem on X. Thinking Machines ran an actual test: they sent "Tell me about Richard Feynman" to Qwen3-235B at temperature 0, repeated it 1000 times, and got 80 unique outputs.

argmax itself is deterministic. The variable is in how the logits are computed—if the numerical values differ even slightly, argmax may choose a different token.

Floating-Point Addition: Order Determines the Result

GPUs store numbers with finite precision. fp16 has only 11 significant bits; when a number falls between 1024 and 2048, there is no room after the decimal point. 1024.4 rounds back to 1024.

Compute 1024 + 0.4 + 0.4:

Left to right: 1024 + 0.4 rounds back to 1024, and adding another 0.4 still gives 1024.

Add the small values first: 0.4 + 0.4 = 0.8; 1024 + 0.8 rounds to 1025.

Each LLM forward pass has billions of such additions. Change the order, and the logits shift slightly.

Why Does the Order Change?

The most common guess is that the completion order of GPU parallel threads is unpredictable. That is inaccurate. A typical forward-pass kernel computes in a fixed order; for the same input and same batch size, matrix multiplication results are bitwise identical.

The real reason is other requests on the server.

The inference server batches multiple users' requests together and runs them at once. The kernel decides how to split the work based on batch size:

• Large batch: every GPU core is busy, and the computation is done in one pass.
• Small batch: some cores are idle, so the kernel splits long sums into chunks, distributes them across cores, and merges the results at the end.

When the split changes, the addition order changes. The batch size is determined by how many other users are using the server at that moment.

Tiny Differences Flip the Entire Answer

Run the same prompt twice:

At batch 8: Queens = 12.3417, New = 12.3415. argmax picks Queens.

At batch 32: Queens = 12.3414, New = 12.3416. argmax picks New.

It only flips when the logits are close to a tie. After the flip, every subsequent token is generated from a different prefix, and the output diverges from there.

In the Thinking Machines test, the first 102 tokens were identical across 1000 runs. At the 103rd token, 992 runs output "Queens, New York" and 8 output "New York City".

Solution: Batch-Invariant Kernels

Make the kernel's splitting method independent of batch size—for example, fix the attention chunk size, even at the cost of parallelism. After Thinking Machines' test, 1000 runs were completely identical.

The cost is speed. On Qwen3-8B, default vLLM processed 1000 requests in 26 seconds; the deterministic version took 55 seconds, and after optimizing the attention kernel, 42 seconds.

The Problem Is Not Just Batch

Commenters added extra variables for multi-GPU scenarios: NCCL all-reduce's algorithm choice depends on message size, and message size is determined by batch size; MoE models distribute tokens to experts via atomic operations, and warp scheduling races affect arrival order; PagedAttention's cache block boundaries change the reduction tree structure.

Someone noticed an even subtler issue: cache hits also change how sums are split. This means that if you run the same prompt twice to verify stability, the conditions for the second run have already been changed by the first.

One commenter's conclusion was pragmatic: either pay roughly 2x the latency for deterministic kernels, or accept nondeterminism and pin eval to a dedicated replica.