What Scales in Cross-Entropy Scaling Law?

TL;DR

The paper investigates why cross-entropy scaling can fail at very large model scales by decomposing cross-entropy into three components based on Rank-based Error ($RBE$): Error-Entropy ($EE$), Self-Alignment ($SA$), and Confidence ($Conf$). It shows theoretically and empirically that only $EE$ exhibits robust power-law scaling with model size, while $SA$ and $Conf$ remain largely invariant, explaining why the traditional cross-entropy scaling law breaks down as the $EE$ share diminishes. The findings introduce the Error-Entropy Scaling Law as a more accurate description of model behavior and offer a practical perspective for training and evaluating large language models. These insights have potential implications for training objectives and the theoretical understanding of AI systems.

Abstract

The cross-entropy scaling law has long served as a key tool for guiding the development of large language models. It shows that cross-entropy loss decreases in a predictable power-law rate as the model size increases. However, recent evidence indicates that this law breaks down at very large scales: the loss decreases more slowly than expected, which causes significant trouble for developing large language models. In this paper, we hypothesize that the root cause lies in the fact that cross-entropy itself does not truly scale; instead, only one of its hidden components does. To investigate this, we introduce a novel decomposition of cross-entropy into three parts: Error-Entropy, Self-Alignment, and Confidence. We show both theoretically and empirically that this decomposition precisely captures the training dynamics and optimization objectives. Through extensive experiments on multiple datasets and 32 models spanning five orders of magnitude in size, we find that only error-entropy follows a robust power-law scaling, while the other two terms remain largely invariant. Moreover, error-entropy constitutes the dominant share of cross-entropy in small models but diminishes in proportion as models grow larger. This explains why the cross-entropy scaling law appears accurate at small scales but fails at very large ones. Our findings establish the error-entropy scaling law as a more accurate description of model behavior. We believe it will have wide applications in the training, understanding, and future development of large language models.

Figures (8)

• Figure 1: Decomposition of the Cross-Entropy Loss. The upper panel illustrates the cross-entropy loss. The lower panel presents the three components decomposed from cross-entropy. The blue curve denotes the Rank-based Error (RBE) distribution, and the orange curve represents the distribution of the ground-truth scores. Error-Entropy pushes the ground-truth tokens towards higher ranks. Self-Alignment aligns the probability score distribution with RBE distribution. And Confidence term increases the norm of the probability score.
• Figure 2: Overview of the decomposition. Rank-based Error (RBE) is the ranking of the ground-truth token. Model's predictions are grouped based on RBE values. For each group, we compute its proportion $p_e$ (termed RBE distribution), the normalized prediction score $q_e$, and the norm of scores $C$. Based on these definitions, cross-entropy can be mathematically decomposed into three components: Error-Entropy, Self-Alignment, and Confidence.
• Figure 3: Evolution of cross-entropy and its decomposed components during training on three datasets. All components are indeed optimized during training, as suggested by the mathematical deduction: Error-Entropy steadily decreases, Self-Alignment declines in the end, and Confidence term increases. The shaded regions highlight the training progress with the most rapid metric changes.
• Figure 4: Evolution of RBE distribution $p_e$ during training. It shows that RBE distribution shifts from nearly uniform to concentrated, thereby minimizing Error-Entropy.
• Figure 5: Output score distribution when ground-truth tokens are ranked at 2 and 4, respectively. Scores drop for tokens ranked lower than ground-truth tokens, thereby increasing Confidence term.