Temperature set to 0, answers can still be altered by requests on the same GPU

Setting temperature to 0 does not mean the output is reproducible. Avi Chawla laid out a very concrete chain on X: the same prompt can produce different logits depending on which other requests are in the same GPU batch.

The key is that floating-point addition is not associative. Many steps in LLM forward computation perform accumulation; matmul, RMSNorm, and attention all reduce thousands of floating-point values into one output. Each addition rounds to the nearest representable value, so the low bits of (a + b) + c and a + (b + c) may differ.

He gave an fp16 example. fp16 has an 11-bit significand; between 1024 and 2048, the spacing between adjacent representable values is 1.0. Computing 1024 + 0.4 + 0.4 from left to right, each addition rounds back to 1024. If you add the two small numbers first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds to 1025. The numerical path changes, so the result changes.

Batch size changes the accumulation order because kernels try to fully utilize the GPU. An H100 has 132 compute units; when the batch is small, each unit does not get enough work. The kernel splits long accumulations into chunks, runs them in parallel on different units, and combines the partial results. Different batch sizes lead to different splitting schemes, different accumulation orders, and therefore tiny differences in logits.

Normally this difference is negligible. Trouble arises when two token scores are nearly tied. Avi Chawla's example: with batch size 8, “Queens” might be 12.3417 and “New” 12.3415; with batch size 32, rounding might turn them into 12.3414 and 12.3416. Under temperature 0, argmax would pick “Queens” in the first case and “New” in the second. The subsequent tokens are all generated based on different prefixes, so the entire completion diverges.

The nondeterminism comes from forward computation; the sampler is not to blame. Temperature 0 only guarantees that argmax is followed within the current forward computation; there is no guarantee for numerical paths across requests.

The fix is to make every reduction batch invariant: regardless of how many rows are in the batch, each row is reduced in a fixed order. The cost is throughput. The kernel can no longer choose the optimal split for each batch size; it must use one fixed strategy, even if some compute units sit idle.

Several major inference engines now provide switches. vLLM has VLLM_BATCH_INVARIANT=1, still in beta. SGLang has --enable-deterministic-inference, which brings a 25% to 45% slowdown. DeepSeek's DeepInfer has DEEPINFER_BATCH_INVARIANT=1.

For evals, RL rollouts, or any task that needs reproducible output, you can turn it on and record the batch composition in your experiment setup. Production is more complicated. Under continuous batching, the server is constantly adding and removing requests at every decoding step, so the batch around you is not fixed.

Some commenters noted that you can pay this slowdown cost during testing, but in production users want speed. Others asked how to set regression thresholds if yesterday's eval fails today because of someone else's traffic. Another commenter said that when comparing two models on a shared endpoint, what gets measured includes not just the models but also the load at that moment.

Avi Chawla's original post is here: link. The long article he mentioned is here. The related quoted tweet is here.