Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

Abstract

Chinchilla Approach 2 is among the most widely used methods for fitting neural scaling laws. Its parabolic approximation introduces systematic biases in compute-optimal allocation estimates, even on noise-free synthetic data. Applied to published Llama 3 IsoFLOP data at open frontier compute scales, these biases imply a parameter underallocation corresponding to 6.5% of the $3.8\times10^{25}$ FLOP training budget and \$1.4M (90% CI: \$412K-\$2.9M) in unnecessary compute at 50% H100 MFU. Simulated multimodal model misallocations show even greater opportunity costs due to higher loss surface asymmetry. Three sources of this error are examined: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry ($α\neq β$). Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, prone to local minima, and harder to implement. Each concern is shown to be unfounded or addressable, especially when the partially linear structure of the objective is exploited via Variable Projection, enabling unbiased inference on all five loss surface parameters through a two-dimensional optimization that is well-conditioned, analytically differentiable, and amenable to dense, or even exhaustive, grid search. It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations. See https://github.com/Open-Athena/vpnls for details and https://openathena.ai/scaling-law-analysis for other results from this study.

Figures (21)

• Figure 1: Approach 2 misallocation costs extrapolated to $3.8\times10^{25}$ FLOPs. Left: Deadweight Compute Loss (DCL) as a percentage of budget; dollar cost ranges for empirical rows are 90% bootstrap CIs. Right: allocation details including true vs. inferred token counts and model sizes, loss penalty, and dollar cost. "Empirical" rows use Approach 2 power laws fit to digitized Llama 3 IsoFLOP data, evaluated against VPNLS and Approach 3 surfaces. "Simulated" rows use synthetic IsoFLOP data generated from each published surface with the XS grid and $3\times$ drift bias used in the Drifting Bias simulation section (Section \ref{['sec:drifting_bias']}).
• Figure 2: Effect of progressive quality control filtering on Approach 2 Deadweight Compute Loss, measured against an Approach 3 surface fit to unfiltered Llama 3 data. Each row cumulatively applies one QC stage. Nearly all DCL reduction comes from the off-center and weak-curvature filters, which target specific Approach 2 biases. The right table reports points and budgets removed at each stage, along with DCL as a percentage of the $3.8\times10^{25}$ FLOP evaluation budget and dollar cost.
• Figure 3: Approach 2 applied to a symmetric loss surface. Left: IsoFLOP curves with fitted parabolas. True ($\times$) and inferred ($+$) optima are indistinguishable. Right: Power-law fit recovers the exact scaling exponent.
• Figure 4: Approach 2 on asymmetric loss surfaces. Note the visible gap between true (dashed) and inferred (solid) power-law lines in the Asymmetric case. The exponents match perfectly, but the intercepts differ.
• Figure 5: Relative error in compute-optimal token prediction when extrapolating from the training range ($10^{17}$--$10^{21}$ FLOPs) to $10^{24}$ FLOPs. Negative values indicate underestimation: the inferred scaling law predicts fewer tokens than optimal. Bars are grouped by sampling grid width. Annotations for the Chinchilla surface show $D^*$ (true compute-optimal token count) versus $\hat{D}^*$ (the Approach 2 estimate); the Small and Large grid annotations are emphasized (thicker borders) as they fall within the realistic 1--2 decade range typical of scaling law experiments, while Extra Small and Extra Large bracket either side as more extreme configurations.