Numerical Error Analysis of Large Language Models

TL;DR

This work provides a rigorous forward error analysis for the forward pass of decoder‑style transformers under finite precision. By deriving Jacobian‑based condition numbers for layer normalization, two‑layer perceptrons, and self‑attention, it aggregates into a depth‑dependent bound showing relative errors can grow exponentially with the number of blocks, with growth tied to the conditioning of $W^{ op}W_q$ and related norms. The authors also present a detailed deterministic rounding framework that yields explicit per‑layer error terms and a main theorem bounding the output error of a deep transformer, corroborated by numerical experiments that reveal mean‑versus‑median discrepancies and practical mitigation strategies such as higher‑precision computation for attention components. The results yield concrete guidelines for stable transformer inference, including precision management and conditioning controls, with potential extensions to mixed‑precision schemes and backward/ training analysis. Overall, the paper provides quantitative insights into how finite‑precision effects propagate through transformer forward passes and how to design more robust, numerically stable LLM inference pipelines.

Abstract

Large language models based on transformer architectures have become integral to state-of-the-art natural language processing applications. However, their training remains computationally expensive and exhibits instabilities, some of which are expected to be caused by finite-precision computations. We provide a theoretical analysis of the impact of round-off errors within the forward pass of a transformer architecture which yields fundamental bounds for these effects. In addition, we conduct a series of numerical experiments which demonstrate the practical relevance of our bounds. Our results yield concrete guidelines for choosing hyperparameters that mitigate round-off errors, leading to more robust and stable inference.

Figures (4)

• Figure 1: Relative round-off errors of applying a deep transformer \ref{['eq:transformer_deep']}. We simulate low-precision computations by rounding intermediate values to various numbers of digits. We set $d, n, D = 20$ and $L = 40$, generate the self-attention matrices $\mathrm{W}_\mathrm{q}, \mathrm{W}_{}, \mathrm{W}_\mathrm{v}$ with centred normally distributed entries of unit variance, and the matrices of the two-layer perceptron with centred normally distributed entries with variance $1/\sqrt{d}$. To increase the condition number of $\mathrm{W}_{}^\intercal \mathrm{W}_\mathrm{q}$, we multiply it on both sides with diagonal matrices with uniform random entries in $[1/4,4]$. The input vectors are initialised with centred normally distributed entries of unit variance. We compute the statistics over $5000$ initializations. We plot in the upper row the mean relative errors in each layer: the almost linear curves indicate close to exponential growth. To visualize the stochastic behaviour of the errors, we plot in the lower row the median error and shade the area between the 5th and 95th percentiles. A histogram of the error distribution in the final layer shows that most of the errors are orders of magnitude lower than the mean error (vertical black lines).
• Figure 2: Relative round-off errors of applying a deep transformer \ref{['eq:transformer_deep']}. We simulate low-precision computations by rounding intermediate values to various numbers of digits. The same experiment as in Figure is performed with $L = 10, 15, 20$ layers, and the relative errors in the last layer are shown. The matrix $\mathrm{W}_{}^\intercal \mathrm{W}_\mathrm{q}$ is additionally multiplied with $\lambda$ to obtain a range of values of $\| \mathrm{W}_{}^\intercal \mathrm{W}_\mathrm{q} \|_{\mathrm{2,2}}\sim \lambda$. The $x$-axis corresponds to $\lambda$. Both componentwise and normwise errors exhibit roughly linear growth with $\lambda$ when $L = 10$, and roughly quadratic growth when $L = 20$.
• Figure 3: Relative round-off errors of applying of applying the self-attention \ref{['eq:attention']}. We simulate low-precision computations by rounding intermediate values to various numbers of digits. We set $d, n = 10$. The matrices $\mathrm{W}_\mathrm{q}, \mathrm{W}_{}, \mathrm{W}_\mathrm{v}$ are identity matrices, and the input $\mathrm{X}$ is generated with normally distributed entries with mean $1$ and variance $0.01$. The displayed results are mean values over 1000 initialisations, one standard deviation is indicated by the shaded area. A slope triangle with slope 2 is shown in grey, indicating quadratic growth.
• Figure 4: Relative round-off errors of applying a deep transformer a) as in \ref{['eq:transformer_deep']} and b) by putting the layer normalization immediately after, instead of before, the self-attention. We simulate low-precision computations by rounding intermediate values to various numbers of digits. We set $d, n, D = 10$. All of the transformer matrices are generated with centred normally distributed entries with variance $0.1$. The input vectors are initialised with centred normally distributed entries of unit variance. The displayed results are mean values over 1000 initialisations, one standard deviation is indicated by the shaded area.

Theorems & Definitions (76)

• Lemma 3.1
• proof
• Remark 3.2
• Lemma 3.3
• proof
• Lemma 3.4
• proof
• Lemma 3.5
• proof
• Lemma 3.6