On the origin of neural scaling laws: from random graphs to natural language

TL;DR

It is demonstrated that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations, and preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization is provided.

Abstract

Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, compute, and number of model parameters. This has spurred an intense interest in the origin of neural scaling laws, with a common suggestion being that they arise from power law structure already present in the data. In this paper we study scaling laws for transformers trained to predict random walks (bigrams) on graphs with tunable complexity. We demonstrate that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations. We further consider dialing down the complexity of natural language systematically, by training on sequences sampled from increasingly simplified generative language models, from 4,2,1-layer transformer language models down to language bigrams, revealing a monotonic evolution of the scaling exponents. Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erdös-Renyi and scale-free Barabási-Albert ensembles. Finally, we revisit conventional scaling laws for language modeling, demonstrating that several essential results can be reproduced using 2 layer transformers with context length of 50, provide a critical analysis of various fits used in prior literature, demonstrate an alternative method for obtaining compute optimal curves as compared with current practice in published literature, and provide preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization.

Figures (23)

• Figure 1: Scaling laws for 2-layer transformers trained on next token prediction on unbiased random walks on an Erdös-Renyi graph, with $8$K nodes and $50$K edges. Neither the random walks nor the graph exhibits any power laws. Data in bottom two plots are fit to Eq. \ref{['1dlossfit']}. Mean $\overline{\alpha_D} = 1.028$ with standard deviation $0.129$. Mean $\overline{\beta_N} = 0.749$ with standard deviation $0.014$. Average MSE for $L(N)_D$ 1d power law fits is $1.07\times 10^{-8}$, compared to $7.03 \times 10^{-8}$ for best exponential fit. Average MSE for $L(D)_N$ 1d power law fits is $8.86 \times 10^{-8}$, compared to $2.13 \times 10^{-6}$ for best exponential fit. Brackets indicate 95% confidence intervals obtained from bias-corrected and accelerated bootstrap method (see Appendix \ref{['app:fitting']} for details).
• Figure 2: Left: Mean exponents $\overline{\alpha_D}$ and $\overline{\beta_N}$ for all experiments reported in this paper. Legend is in the format <dataset> (model). Language refers to Fineweb-edu. $\overline{\alpha_D}$ and $\overline{\beta_N}$ are averages of the best fit exponents $\alpha_D$ and $\beta_N$ over $D$ and $N$. Error bars indicate standard deviation of the best fits $\alpha_D$, $\beta_N$ across different $N$ and $D$ respectively. Right: $\overline{\alpha_D}$ and $\overline{\beta_N}$ for 2-layer transformer experiments on language, T1L, T2L, and language bigrams, demonstrating monotonic evolution of $\overline{\alpha_D}$ with approximate entropy of the dataset, along with relative stability of $\overline{\beta_N} \approx 0.5$.
• Figure 3: Stationary distribution on nodes $p(v)$ (unigram distribution) and transition probability distributions $p(u|v)$ (bigram distribution) plotted against the rank for various representative cases studied in this paper. Note that $p(v)$ is equal (up to an overall constant) to the degree distribution of the graph when $\kappa = 0$. Left two columns are plotted on log-linear scale, and the right two columns are on log-log scale. ER $\kappa = 0$ case is tightly concentrated around its mean probability, as indicated by roughly constant size plateaus. BA ($\kappa = 0)$) has power laws in both unigram and bigram distributions; the unigram exponent is approximately equal to the theoretical expectation $1/(\gamma - 1) = 0.5$. The power law in the bigram is more clear if plotted in reverse rank order (not shown), but can be observed from the decreasing width of the plateaus. Language results are from Fineweb-edu using GPT-2 tokenizer; unigram distribution is a standard Zipf plot, while bigram distribution shows broken power law.
• Figure 4: Scaling laws for Erdos-Renyi graph with 8K nodes and 50K edges, with power-law transition probabilities set by $\kappa = 1$. Mean exponent $\overline{\alpha_D} = 0.925$ with standard deviation $0.084$. Mean exponent $\overline{\beta_D} = 0.741$, with standard deviation $0.010$. Average MSE for $L(N)_D$ 1d power law fits is $1.76 \times 10^{-8}$, compared to $1.03 \times 10^{-7}$ for best exponential fit. Average MSE for $L(D)_N$ 1d power law fits: $5.15 \times 10^{-8}$, compared to $2.12 \times 10^{-6}$ for best exponential fit.
• Figure 5: Scaling laws for unbiased ($\kappa = 0$) random walk on Erdos-Renyi graph with 1K nodes and 5K edges. Mean exponent $\overline{\alpha_D} = 0.665$ with standard deviation $0.125$. Mean exponent $\overline{\beta_D} = 0.684$, with standard deviation $0.021$. Average MSE for $L(N)_D$ 1d power law fits is $2.41e-08$, compared to $1.96 \times 10^{-7}$ for best exponential fit. Average MSE for $L(D)_N$ 1d power law fits: $3.37 \times 10^{-8}$, compared to $1.97 \times 10^{-6}$ for best exponential fit.