Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

TL;DR

The paper reveals that LLM inference reproducibility is undermined by floating-point precision and hardware/config variations, showing that greedy decoding can be non-deterministic and that sampling-based evaluations inherit precision-driven variance. It provides a thorough, controlled analysis across models, tasks, and GPUs, quantifying how BF16 induces output divergence and longer response lengths, and demonstrates that FP32 largely mitigates these issues. To address the problem, the authors introduce LayerCast, a memory-efficient pipeline that computes in FP32 while storing weights in BF16, achieving near FP32 determinism with substantial memory savings. The work advocates standardized evaluation practices that explicitly account for numerical-precision effects and provides a practical path to reproducible inference in real-world deployments.

Abstract

Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.

Figures (9)

• Figure 1: Left: Under BF16 precision and greedy decoding, the model's output can vary significantly depending on factors such as GPU count, evaluation batch size, and GPU hardware version. Right: For example, changes in evaluation batch size alone can lead to noticeable differences in responses, which is often ignored and not standardized by evaluation benchmarks.
• Figure 2: Floating-point format of FP32, FP16 and BF16.
• Figure 3: Left: the top-5 tokens and their predicted probabilities at the divergence index for two different answers to the same question in BF16. Right: The gap between the top-two competing tokens probability. We observe the token probability gap are often minimal in reasoning models.
• Figure 4: Avg_Std@top1_prob across 12 different settings for 4 models and 5 tasks, under BF16, FP16 and FP32. FP16 shows significantly lower variance compared to BF16. FP32 yields near-zero variance, demonstrating strong robustness to floating-point rounding errors.
• Figure 5: Distribution of Div_Index for DeepSeek-R1-Distill-Qwen-7B on MATH500 under BF16, FP16, and FP32. Higher numerical precisions lead to fewer divergent examples and a shift of divergence point to later token positions.