Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

Xuan Tang, Jichu Li, Difan Zou

TL;DR

This work introduces the first convergence framework for adaptive optimizers under hardware-aware floating-point quantization that includes gradients, weights, and optimizer states. It provides quantized convergence guarantees for Adam and Muon on smooth non-convex objectives, showing that rates close to full-precision can be preserved when mantissa length grows only as $M=\Theta(\log T)$. Adam is shown to be particularly sensitive to weight and second-moment quantization due to $\beta_2\to1$, while Muon tolerates weaker relative errors thanks to its matrix-based sign operator, explaining its empirical robustness. The results bridge theory and practice for low-precision training in large models, and experiments on synthetic and real data corroborate the theoretical bounds. This framework informs precision choices and optimizer design for efficient, scalable low-precision optimization in modern deep learning systems.

Abstract

The rapid scaling of large language models (LLMs) has made low-precision training essential for reducing memory, improving efficiency, and enabling larger models and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware-aware quantization, leaving open the question of why low-precision training remains effective. We introduce the first theoretical framework for analyzing the convergence of adaptive optimizers, including Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quantization errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis further reveals that Adam is highly sensitive to weights and second-moment quantization due to its reliance on $β_2 \to 1$, while Muon requires weaker error control and is thus potentially more robust. These results narrow the gap between empirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.

A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

TL;DR

This work introduces the first convergence framework for adaptive optimizers under hardware-aware floating-point quantization that includes gradients, weights, and optimizer states. It provides quantized convergence guarantees for Adam and Muon on smooth non-convex objectives, showing that rates close to full-precision can be preserved when mantissa length grows only as . Adam is shown to be particularly sensitive to weight and second-moment quantization due to , while Muon tolerates weaker relative errors thanks to its matrix-based sign operator, explaining its empirical robustness. The results bridge theory and practice for low-precision training in large models, and experiments on synthetic and real data corroborate the theoretical bounds. This framework informs precision choices and optimizer design for efficient, scalable low-precision optimization in modern deep learning systems.

Abstract

The rapid scaling of large language models (LLMs) has made low-precision training essential for reducing memory, improving efficiency, and enabling larger models and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware-aware quantization, leaving open the question of why low-precision training remains effective. We introduce the first theoretical framework for analyzing the convergence of adaptive optimizers, including Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quantization errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis further reveals that Adam is highly sensitive to weights and second-moment quantization due to its reliance on , while Muon requires weaker error control and is thus potentially more robust. These results narrow the gap between empirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.
Paper Structure (63 sections, 23 theorems, 231 equations, 11 figures, 3 algorithms)

This paper contains 63 sections, 23 theorems, 231 equations, 11 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. This paper.
  3. Notations.
  4. Related Work
  5. Adaptive Optimization.
  6. Low-bit Training.
  7. Quantization Convergence.
  8. Preliminaries and Problem Setup
  9. Preliminaries
  10. Floating-point quantization.
  11. Relative error.
  12. Problem Setup
  13. Low-precision training framework.
  14. Relative errors.
  15. Optimizers.
  16. ...and 48 more sections

Key Result

Theorem 4.5

Suppose Assumptions assump:qe, assump:grad_unbiased–assump:bound_init hold. Let $d=mn$ be the number of trainable parameters, consider the Quantized Adam algorithm defined in alg:framework run for $T$ iterations with $\eta_t = (1 - \beta_1) \Omega_t \eta$, where $\Omega_t = \sqrt{\sum_{j=0}^{t-1} \b where $C$ is a constant depending on the problem hyperparameters, and $\tilde{Q}(T)$ is a function

Figures (11)

  • Figure 1: Floating-point quantization from fp32 to bf16. Only the mantissa is truncated, while sign and exponent remain unchanged.
  • Figure 2: An analytical low-precision training framework
  • Figure 3: Rosenbrock: Adam gradient norms under different mantissa precisions $M$ (left: full $10{,}000$ iterations; right: last $100$ iterations). Larger mantissa bit-lengths yield smaller converged gradient norms. Together with Figure \ref{['fig:Adam_rosenbrock_qe']}, this shows that higher precision reduces quantization error and improves convergence, consistent with Theorem \ref{['thm:QAdam']}.
  • Figure 4: Rosenbrock: Muon gradient norms under different mantissa precisions $M$ (left: full $10{,}000$ iterations; right: last $100$ iterations). Larger mantissa bit-lengths yield smaller converged gradient norms. Together with Figure \ref{['fig:Muon_rosenbrock_qe']}, this shows that higher precision reduces quantization error and improves convergence, consistent with Theorem \ref{['thm:QMuon']}.
  • Figure 5: Rosenbrock: Adam relative quantization error of different mantissa bits ($M$). Weights error (top left), Gradient error (top right), First moment error (bottom left), Second moment error (bottom right). These results show that the more mantissa bits, the smaller the relative quantization error. Combining with Figure \ref{['fig:Adam_rosenbrock']}, we can see that the more mantissa bits, the smaller quantization error, the better convergence performance (Theorem \ref{['thm:QAdam']}).
  • ...and 6 more figures

Theorems & Definitions (45)

  • Theorem 4.5: Convergence of Quantized Adam
  • Theorem 4.6: Convergence of Quantized Muon
  • Lemma A.1: Equivalence of Perturbed Dynamical Systems
  • proof
  • Lemma A.2: The value range of $v_{t,i}$ and the upper bound of $|\frac{1}{\sqrt{\epsilon+v_{t,i}}}-\frac{1}{\sqrt{\epsilon+v'_{t,i}}}|$
  • proof
  • Lemma A.3: Bound on Discrete Error
  • proof
  • Lemma A.4: Finite Geometric Series Ratio Bounded by Infinite Sum
  • proof
  • ...and 35 more