Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Laws for Precision in High-Dimensional Linear Regression

Dechen Zhang, Xuan Tang, Yingyu Liang, Difan Zou

TL;DR

This work rigorously characterizes the complex interplay among model scale, dataset size, and quantization error, and provides a principled theoretical basis for optimizing training protocols under practical hardware constraints.

Abstract

Low-precision training is critical for optimizing the trade-off between model quality and training costs, necessitating the joint allocation of model size, dataset size, and numerical precision. While empirical scaling laws suggest that quantization impacts effective model and data capacities or acts as an additive error, the theoretical mechanisms governing these effects remain largely unexplored. In this work, we initiate a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework. By analyzing multiplicative (signal-dependent) and additive (signal-independent) quantization, we identify a critical dichotomy in their scaling behaviors. Our analysis reveals that while both schemes introduce an additive error and degrade the effective data size, they exhibit distinct effects on effective model size: multiplicative quantization maintains the full-precision model size, whereas additive quantization reduces the effective model size. Numerical experiments validate our theoretical findings. By rigorously characterizing the complex interplay among model scale, dataset size, and quantization error, our work provides a principled theoretical basis for optimizing training protocols under practical hardware constraints.

Scaling Laws for Precision in High-Dimensional Linear Regression

TL;DR

This work rigorously characterizes the complex interplay among model scale, dataset size, and quantization error, and provides a principled theoretical basis for optimizing training protocols under practical hardware constraints.

Abstract

Low-precision training is critical for optimizing the trade-off between model quality and training costs, necessitating the joint allocation of model size, dataset size, and numerical precision. While empirical scaling laws suggest that quantization impacts effective model and data capacities or acts as an additive error, the theoretical mechanisms governing these effects remain largely unexplored. In this work, we initiate a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework. By analyzing multiplicative (signal-dependent) and additive (signal-independent) quantization, we identify a critical dichotomy in their scaling behaviors. Our analysis reveals that while both schemes introduce an additive error and degrade the effective data size, they exhibit distinct effects on effective model size: multiplicative quantization maintains the full-precision model size, whereas additive quantization reduces the effective model size. Numerical experiments validate our theoretical findings. By rigorously characterizing the complex interplay among model scale, dataset size, and quantization error, our work provides a principled theoretical basis for optimizing training protocols under practical hardware constraints.
Paper Structure (92 sections, 83 theorems, 560 equations, 2 figures)

This paper contains 92 sections, 83 theorems, 560 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Our setting.
  3. Notations.
  4. Our main results.
  5. Related Work
  6. Empirical scaling laws for quantized training.
  7. High-dimensional linear regression via SGD.
  8. Theoretical understandings of scaling laws.
  9. Theoretical Setup
  10. Quantization Operation
  11. Data Model
  12. Main Theory
  13. Multiplicative Quantization
  14. Additive Quantization
  15. Connection with empirical scaling laws for low-precision training.
  16. ...and 77 more sections

Key Result

Theorem 4.1

Suppose $\gamma<\frac{1}{(1+\overline{\epsilon}_{2}^{(M)})\alpha\mathrm{tr}(\mathbf{H}_f^{(q)})}$. For any $i\in \{s,d,f,p,a,o\}$, if there exist $\overline{\epsilon}_i$ such that quantization $\mathcal{Q}_i$ is $\overline{\epsilon}_i$-multiplicative, then under Assumption ass1:unbiased, ass: data c where $M_{\rm eff}=M$ and $N_{\rm eff}=N\left[\frac{1+\overline{\epsilon}_2^{(M)}}{(1-\overline{\ep

Figures (2)

  • Figure 1: Scaling of excess risk $\mathbb{E}[\mathcal{R}] - \frac{1}{2}\sigma^2$ under multiplicative quantization with $\epsilon = 10^{-3}$, $\gamma = 0.1$, $\sigma = 1$. (a), (b): $a = 1.5$, $p = 10{,}000$; (c), (d): $a = 2.0$, $p = 1{,}000$. Panels (a), (c) fix $M_{\mathrm{eff}}$ and vary $N_{\mathrm{eff}}$; panels (b), (d) fix $N_{\mathrm{eff}}$ and vary $M_{\mathrm{eff}}$. Fitted exponents (orange curves) match theoretical predictions: $\alpha = -(a-1)$ and $\beta = -(a-1)/a$. All fits achieve $R^2 > 0.99$.
  • Figure 2: Scaling of excess risk $\mathbb{E}[\mathcal{R}] - \frac{1}{2}\sigma^2$ under additive quantization with $\epsilon = 10^{-8}$, $\gamma = 0.1$, $\sigma = 1$. (a), (b): $a = 1.5$, $p = 10{,}000$; (c), (d): $a = 2.0$, $p = 1{,}000$. Panels (a), (c) fix $M_{\mathrm{eff}}$ and vary $N_{\mathrm{eff}}$; panels (b), (d) fix $N_{\mathrm{eff}}$ and vary $M_{\mathrm{eff}}$. Fitted exponents (orange curves) match theoretical predictions: $\alpha = -(a-1)$ and $\beta = -(a-1)/a$. All fits achieve $R^2 > 0.99$.

Theorems & Definitions (164)

  • Definition 3.1
  • Theorem 4.1: Scaling law under multiplicative quantization, an upper bound
  • Theorem 4.2: Scaling law under additive quantization, an upper bound
  • Theorem 4.3: Scaling law under multiplicative quantization, a lower bound
  • Theorem 4.4: Scaling law under additive quantization, a lower bound
  • Lemma 5.1: Excess risk bounds under general quantization
  • Lemma 5.2: Excess risk bounds under multiplicative quantization
  • Lemma 5.3: Bounds under polynomial spectrum, multiplicative quantization
  • Lemma 5.4: Bounds under polynomial spectrum, additive quantization
  • proof
  • ...and 154 more