Relative-Based Scaling Law for Neural Language Models

TL;DR

The paper addresses the incomplete picture provided by cross-entropy in scaling laws by introducing Relative-Based Probability (RBP) and the Relative-Based Scaling Law, which describe how the probability that the ground-truth token appears in the top-$k$ predictions improves with model size. The authors show that for small $k$ (notably $k$ up to a few dozen) the relation $-\,\log \mathrm{RBP}_k \propto S^{-\alpha}$ holds with high fidelity ($R^2$ ~ 0.99) across four datasets and four model families spanning five orders of magnitude. They further demonstrate three practical regimes ($k=1$, moderate $k$, large $k$), analyze emergence phenomena under top-$k$ sampling, and reveal a potential unification with cross-entropy through a lognormal rank-distribution framework, predicting similar scaling slopes for CE and $-\log \mathrm{RBP}_1$. The work provides both practical insights for decoding strategies and theoretical avenues toward a more unified understanding of scaling laws in neural language models.

Abstract

Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of performance: it measures the absolute probability assigned to the correct token, but ignores the relative ordering between correct and incorrect tokens. Yet, relative ordering is crucial for language models, such as in greedy-sampling scenario. To address this limitation, we investigate scaling from the perspective of relative ordering. We first propose the Relative-Based Probability (RBP) metric, which quantifies the probability that the correct token is ranked among the top predictions. Building on this metric, we establish the Relative-Based Scaling Law, which characterizes how RBP improves with increasing model size. Through extensive experiments on four datasets and four model families spanning five orders of magnitude, we demonstrate the robustness and accuracy of this law. Finally, we illustrate the broad application of this law with two examples, namely providing a deeper explanation of emergence phenomena and facilitating finding fundamental theories of scaling laws. In summary, the Relative-Based Scaling Law complements the cross-entropy perspective and contributes to a more complete understanding of scaling large language models. Thus, it offers valuable insights for both practical development and theoretical exploration.

Figures (8)

• Figure 1: Illustration of the limitation of absolute-based metrics in evaluating generation performance. (Left) The ground-truth token has a high absolute probability but is ranked below competitors, leading greedy decoding to fail. (Right) With the same absolute probability yet a higher rank, the ground-truth token is correctly chosen. This shows that absolute-based metrics cannot capture the crucial relative ranking among tokens.
• Figure 2: Illustration of two distinct metric calculation methods. Figure \ref{['fig:absolute_metric']} shows the absolute-based metric $\rm p_A$, which directly quantifies the probability assigned to the ground-truth. Figure \ref{['fig:relative_metric']} shows the relative-based metric $\text{RBP}_{k}$, which measures how often the ground-truth falls within the top-$k$ predictions. These figures highlight how the two metrics capture different aspects of model behavior.
• Figure 3: Relative-based scaling laws when $k=1$. Across all model series and all datasets, $\text{RBP}_1$ exhibits precise power-law relationship with model sizes.
• Figure 4: Scaling laws when $1 < k \ll |\mathcal{V}|$. The relative-based metric $\text{RBP}_{k}$ maintains strong power-law scaling behavior, with consistently high $R^2$ values across datasets and model series. This indicates that the metric reliably captures performance improvements under top-$k$ sampling strategies commonly used in practice.
• Figure 5: Scaling laws when $k \rightarrow |\mathcal{V}|$. In this regime, the power-law behavior deteriorates substantially, with large scatter and inconsistent slopes across models and datasets. This indicates that $\text{RBP}_{k}$ becomes less informative about scaling when the threshold approaches the vocabulary size.