Neural Scaling Laws Rooted in the Data Distribution

TL;DR

The paper investigates why neural networks exhibit power-law scaling of error with model/data size by building a percolation-based model of natural data distributions. It derives two principal scaling regimes—subcritical with a Zipf-like distribution of learnable subtasks (quanta) and supercritical with a dominant data manifold—grounding and unifying prior scaling theories. The authors formalize the theory, connect it to known critical exponents (e.g., $\tau=5/2$, $\alpha=1$, $D=4$ for $d\ge 6$), and validate it in toy experiments using Bethe-lattice percolation with a nearest-neighbor regression model, observing agreement with the predicted model- and data-scaling laws. The framework offers a path toward quantitatively predicting language-model scaling and motivates future empirical tests on natural data distributions near criticality to forecast and guide the scaling of large AI systems.

Abstract

Deep neural networks exhibit empirical neural scaling laws, with error decreasing as a power law with increasing model or data size, across a wide variety of architectures, tasks, and datasets. This universality suggests that scaling laws may result from general properties of natural learning tasks. We develop a mathematical model intended to describe natural datasets using percolation theory. Two distinct criticality regimes emerge, each yielding optimal power-law neural scaling laws. These regimes, corresponding to power-law-distributed discrete subtasks and a dominant data manifold, can be associated with previously proposed theories of neural scaling, thereby grounding and unifying prior works. We test the theory by training regression models on toy datasets derived from percolation theory simulations. We suggest directions for quantitatively predicting language model scaling.

Figures (10)

• Figure 1: Visualization of site percolation on a $10 \times 10 \times 10$ lattice.
• Figure 2: Theoretical scaling law as a function of model DOF $N$, for $\alpha = 1$ and various values of $c/D$.
• Figure 3: Theoretical scaling law as a function of dataset size $\mathcal{D}$, for $\alpha = 1$ and various values of $c/D$.
• Figure 4: Exponent ratios controlling model scaling (top) and data scaling (bottom), meaningful when $p \lesssim p_c$. The overall scaling law regime transitions about a ratio of 1 for both.
• Figure 5: Loss achieved by a function approximator with parameterized DOF distribution for $N = 500$. A black cross indicates the parameters predicted to be optimal for model scaling.