Unraveling the Mystery of Scaling Laws: Part I

TL;DR

This work revisits scaling laws for language models, confirming the original $L(N)=(N_c/N)^{\alpha_N}$ form extends up to $N\approx 33\mathrm{B}$ and providing transparent procedures to estimate all constants from small models ($N$ in the range $1\mathrm{M}$–$60\mathrm{M}$). It derives and links three core functions, $L(N)$, $L(N,S_{min})$, and $L(N,S,B)$, enabling accurate predictions of final loss, required training steps and tokens, and optimal batch-size trade-offs before any large-scale training. The method emphasizes that coefficients, not the exponents, depend on data, context length, and hyperparameters, and shows practical utility by predicting loss trajectories on large models for both in-domain and out-of-domain data. The results support a principled, computation-aware approach to configuring pre-training, including batch-size strategies, data-mix decisions, and budget appropriations, with extensions proposed for context length and mixture-of-experts architectures.

Abstract

Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training, ultimately contributing to the success of large language models such as GPT-4, Llama and Gemini. However, the original scaling law paper by OpenAI did not disclose the complete details necessary to derive the precise scaling law formulas, and their conclusions are only based on models containing up to 1.5 billion parameters. Though some subsequent works attempt to unveil these details and scale to larger models, they often neglect the training dependency of important factors such as the learning rate, context length and batch size, leading to their failure to establish a reliable formula for predicting the test loss trajectory. In this technical report, we confirm that the scaling law formulations proposed in the original OpenAI paper remain valid when scaling the model size up to 33 billion, but the constant coefficients in these formulas vary significantly with the experiment setup. We meticulously identify influential factors and provide transparent, step-by-step instructions to estimate all constant terms in scaling-law formulas by training on models with only 1M~60M parameters. Using these estimated formulas, we showcase the capability to accurately predict various attributes for models with up to 33B parameters before their training, including (1) the minimum possible test loss; (2) the minimum required training steps and processed tokens to achieve a specific loss; (3) the critical batch size with an optimal time/computation trade-off at any loss value; and (4) the complete test loss trajectory with arbitrary batch size.

Figures (3)

• Figure 1: Left: Actual and predicted loss trajectories of a 2B model on the C4 test data (Section \ref{['sec:c4_scale']}). Right: Actual and predicted loss trajectories of a 33B model on the code test data (Section \ref{['sec:mix_scale']}). The actual and predicted loss trajectories closely align, especially after the initial warm-up stage.
• Figure 2: Batch size scan of a 10M model with 4096 context length. Each curve has the same loss value with varying batch sizes and training steps.
• Figure 3: Actual and predicted loss trajectories of 500M, 2B and 33B models on the out-of-domain private Chinese test data (Section \ref{['sec:mix_scale']}). The loss trajectory on out-of-domain test data has large fluctuations, but the overall trend and final converged loss values still closely align with the predictions. The estimated constant values in scaling-law formulas are provided on the bottom right.