Reconciling Kaplan and Chinchilla Scaling Laws

TL;DR

This work tackles the apparent mismatch between Kaplan 2020 and Chinchilla 2022 scaling laws for transformer language models. It develops an analytical framework to convert Chinchilla’s total-parameter and total-compute scaling into Kaplan’s non-embedding basis, showing that embedding parameters and small-scale effects reconcile the previously reported coefficients. The authors demonstrate that the local scaling exponent linking non-embedding parameters to non-embedding compute aligns with Kaplan’s value when accounting for embedding contributions, and they reconcile compute–loss relationships by incorporating offsets and basis changes. They recommend standardizing scaling studies to report total parameters and total compute with an offset in loss–compute modeling to improve cross-study comparability and guide efficient compute allocation.

Abstract

Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters ($N$) and training tokens ($D$) should be set to achieve the lowest possible loss for a given compute budget ($C$). Kaplan: $N_\text{optimal} \propto C^{0.73}$, Chinchilla: $N_\text{optimal} \propto C^{0.50}$. This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Kaplan's original overestimation. As a second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.

Figures (7)

• Figure 1: Overview of the approach used to reconcile the two studies.
• Figure 2: Total parameter count vs. non-embedding parameter count for the suite of models sizes used in the Chinchilla study, along with our fitted approximation. Note the curvature at model sizes below around 200M parameters.
• Figure 3: Visualization of Eq. \ref{['eq_CE_NE']} & \ref{['eq_g']}, using the Epoch AI specification.
• Figure 4: Synthetic training curves from Eq. \ref{['eq_loss_NE_CE']} fitted to the Chinchilla data using the Epoch AI specification. Curves are generated for 20 logarithmically-spaced models matching Kaplan's size range. Left in terms of training tokens, right in terms of non-embedding compute, as used in the Kaplan study. Hence the right plot can be viewed as Chinchilla's loss curves, adjusted to match Kaplan's model sizes and non-embedding measurements.
• Figure 5: Using the synthetic training curves generated in Figure \ref{['fig_chinchilla_synth']}, we empirically fit the frontier of compute efficient models using Chinchilla's Method 1. This gives our main result; synthetic training curves generated from Chinchilla's study, adjusted for model sizes and non-embedding compute, produce a local scaling coefficient ${N^*_{\setminus E}} \propto C_{\setminus E}^{0.78}$, close to Kaplan's reported coefficient of 0.73. The analytical function from Eq. \ref{['eq_CE_NE']} is also verified.