Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism

TL;DR

This work reframes nondeterminism in large language model inference by focusing on token-level probability variations across runs, rather than solely on final text. It defines two metrics, the standard deviation $\sigma_j$ and the range $R_j$, to quantify per-token probability fluctuations and analyzes their behavior across multiple models, batch sizes, and hardware platforms, including Huawei GPUs. The key finding is that nondeterminism is most pronounced for mid-range token probabilities (roughly 0.2–0.8) and grows with batch size, while probabilities near 0 or 1 are largely stable; this pattern is remarkably consistent across models and GPUs. The paper also shows that a single inference analysis of token probabilities can help estimate the potential impact of nondeterminism on downstream tasks, offering a practical tool for evaluating and mitigating variability in real-world deployments.

Abstract

The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results. This is due to the finite precision effects of the arithmetic operations, which depend on the order in which they are executed. This order, in turn, depends on the processes that are running concurrently on the GPU. Previous studies have focused on the impact of nondeterminism on the text generated by the LLMs or on proposing mechanisms to achieve deterministic execution. This work takes a closer look at nondeterminism by analyzing the variations on the token probabilities, not on the generated text. Interestingly, all the models evaluated have similar results in both the trends and the actual values of the variations of the probabilities. In particular, the results show that the effects of nondeterminism are significant for token probabilities that are in the range of 0.1 to 0.9, while they are much smaller when the probabilities are close to 0 or 1. This has significant implications for our understanding of nondeterminism. The first is that nondeterminism will likely have a non-negligible impact on generated text when the temperature is not zero, as it introduces significant variations in the token probabilities except when they are close to 0 or 1. Secondly, it suggests that all models have similar non deterministic variations at the token probability level. Therefore, different variations in the performance of the generated text, for example, when measuring accuracy on a benchmark, seem to come from different token probabilities or response lengths. A third implication is that we may be able to estimate the impact of nondeterminism by running a single inference and analyzing the token level probabilities, instead of having to run the same inference many times.

Figures (10)

• Figure 1: Diagram showing the factors involved in nondeterminism evaluation.
• Figure 2: Distribution of average range $R_j$ and standard deviation $\sigma_j$ of variations in token probability due to nondeterminism for different batch sizes and models across different GPUs: NVIDIA A100 (top), NVIDIA A6000, NVIDIA H200, Huawei Ascend-910
• Figure 3: Distribution of average token probability for different batch sizes and models across different GPUs: NVIDIA A100 (top), NVIDIA A6000, NVIDIA H200, Huawei Ascend-910
• Figure 4: Effect of Softmax in non-determinism. The probability of the first token in a two-token vocabulary calculated by Softmax is identical to the probability of the positive class calculated by the Sigmoid function on the difference of the logits.
• Figure 5: Average Range $R_j$ of variations in token probability due to nondeterminism for different batch sizes and models as a function of the token probability for different GPUs: NVIDIA A100 (top), NVIDIA A6000, NVIDIA H200, Huawei Ascend-910