Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check

TL;DR

The paper investigates whether downstream scaling laws reliably predict large-scale downstream performance from small-scale proxies. It conducts a meta-analysis across 46 tasks and finds that predictable scaling occurs in only 39% of cases, with many tasks showing inverse, nonmonotonic, or breakthrough patterns. It shows that scaling behavior is highly context-dependent, shaped by pretraining data, validation data, and downstream tasks, and these factors can flip or qualitatively change scaling curves. The authors argue for embracing irregular scaling in modeling large-scale models and for rigorous, setup-specific verification of scaling laws, highlighting the need for theory to explain when predictable scaling arises.

Abstract

Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.

Figures (12)

• Figure 1: Revisiting the 46 tasks studied in gadre2025language, we find that only 18 tasks---or 39%---demonstrate smooth, predictable improvement (Figure \ref{['fig:scaling-behaviors_predictable']}). The other 28 tasks are shown in Figures \ref{['fig:scaling-behaviors_inverse']} through \ref{['fig:scaling-behaviors_breakthrough']}, where we group them into different degenerate scaling behaviors: inverse, nonmonotonic, noisy, trendless, and breakthrough scaling. See Figure \ref{['fig:scaling-behaviors']} for examples.
• Figure 2: A taxonomy of different scaling behaviors. Predictable scaling fits closely to a linear functional form after, for example, exponentiating the cross-entropy loss. However, depending on the downstream task, models do not always improve with scale (inverse, nonmonotonic, and trendless), or the improvement might be highly noisy. The improvement might also follow a functional form that is difficult to extrapolate like a sigmoid (breakthrough).
• Figure 3: Choosing a different validation corpus can exaggerate or even reverse which pretraining corpus appears superior. On HellaSwag, the C4 corpus seems better than RedPajama when using 100 PLs as the validation set. Conversely, the scaling trends on CoQA for C4 and RedPajama flip when computing validation perplexity on C4 versus 100 PLs.
• Figure 4: Scaling behavior changes depending on the experimental setting. gadre2025language and magnusson2025datadecidepredictbestpretraining both train language models on C4 and evaluate on MMLU, BoolQ, and Commonsense QA. Still, they differ in their details, such as model architecture, task formatting, or the number of answer choices (in the case of Commonsense QA). Even with the same corpora and downstream task, scaling trends can be dramatically different.
• Figure 5: The 18 tasks in gadre2025language with scaling behavior well-described by a linear scaling law after transforming the cross-entropy loss.