Neural Neural Scaling Laws

TL;DR

Traditional scaling laws model training loss as a fixed parametric form, but downstream task performance exhibits diverse scaling behaviors that are not captured by such fits. NeuNeu reframes downstream performance prediction as time-series extrapolation that jointly leverages token-level validation losses and observed accuracy trajectories, using a LossEncoder, Transformer, and Quantile-Regression head to produce calibrated predictions and uncertainty. Trained on open-source HuggingFace trajectories across multiple model sizes and 66 downstream tasks, NeuNeu achieves a mean absolute error of $2.04\%$, a 38% improvement over logistic scaling laws, and demonstrates strong zero-shot generalization and competitive ranking accuracy. These results suggest neural scaling laws can more accurately forecast downstream performance, enabling better resource allocation and offering a foundation-model view of training dynamics with practical in-silico experimentation potential.

Abstract

Neural scaling laws predict how language model performance improves with increased compute. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors: some improve monotonically, others plateau, and some even degrade with scale. We argue that predicting downstream performance from validation perplexity suffers from two limitations: averaging token-level losses obscures signal, and no simple parametric family can capture the full spectrum of scaling behaviors. To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without assuming any bottleneck or functional form. Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 2.04% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 38% reduction compared to logistic scaling laws (3.29% MAE). Furthermore, NeuNeu generalizes zero-shot to unseen model families, parameter counts, and downstream tasks. Our work suggests that predicting downstream scaling laws directly from data outperforms parametric alternatives.

Figures (12)

• Figure 1: Richer signal from token-level losses (center) enables NeuNeu to better forecast accuracies for downstream tasks (right). Average validation loss, used in logistic scaling laws, averages away token-level loss changes.
• Figure 2: NeuNeu encodes and processes token-level validation probabilities alongside a sequence of historical downstream accuracies and compute gaps, which are projected into context tokens. The BERT-style Transformer devlin-etal-2019-bert backbone uses this information to predict a distribution over the downstream accuracy via quantile regression on the [CLS] token.
• Figure 3: Generalization results for downstream task accuracy prediction.
• Figure 4: NeuNeu is the best predictor of downstream performance. Black dots are ground truth accuracies for the training run, and the grey line marks the beginning of NeuNeu's predictions, after observing the first 20% of downstream accuracies. The light-green band is the 10%-90% interquantile range predicted by NeuNeu itself.
• Figure 5: Neural models are better predictors than logistic scaling laws, even on tasks they have never seen during training.