Quiet Feature Learning in Algorithmic Tasks

TL;DR

This work investigates how Transformer-based language models acquire algorithmic capabilities by training on ten foundational tasks and tracking scaling-law-like behavior. It reveals two phases of learning—a slow phase with little loss improvement and a fast phase with abrupt gains—and shows that quiet features, representing intermediate computations, emerge during the slow phase before any loss drop. Through linear probing and targeted ablations, the authors demonstrate that these quiet features are causally necessary for later success, challenging the notion that loss improvements fully reflect representational progress. The findings suggest the need for richer diagnostics when monitoring model training and imply that substantial internal reorganization can precede observable capability gains, with implications for evaluating and guiding learning in larger models. The study computes scaling laws over budgets from $10^9$ to $10^{15}$ FLOPs, analyzes residual-stream representations, and uses causal ablations to show the hidden progression toward algorithmic competence.

Abstract

We train Transformer-based language models on ten foundational algorithmic tasks and observe pronounced phase transitions in their loss curves that deviate from established power-law scaling trends. Over large ranges of compute, the validation loss barely improves, then abruptly decreases. Probing the models' internal representations reveals that quiet features are learned prior to any decrease in task loss. These quiet features represent intermediate algorithmic computations that do not by themselves improve the output loss. Ablation experiments demonstrate that individual quiet features are causally necessary for task performance. Our results demonstrate that substantial representational progress can remain hidden beneath an apparently flat loss curve, challenging the prevailing use of cross-entropy as a proxy for learning and motivating richer diagnostics for monitoring model training.

Figures (9)

• Figure 1: Model performance (validation loss) abruptly improves as we increase the model size, dataset size, and amount of compute (Training FLOPs) used for training. The input size for addition, multiplication and activity selection is 16. For graph tasks, the input size is 11. For maximum subarray, the input size is 64 while for majority of majorities it is 32. The red dotted line indicates random performance.
• Figure 2: Model exhibit similar abrupt improvement in performance during a single training run. Plots show compute-optimal training runs for the smallest compute budget where test accuracy is 100%. The red dotted line indicates random performance.
• Figure 3: Models learn quiet features before the phase transition. The loss is averaged over the first third of token positions (Beginning), second third (Middle), and last third (End). The red vertical line indicates the task success threshold, which is the smallest compute budget at which the task loss starts to decrease (see Technical Appendix Figure 6). Horizontal dotted lines represent random baselines.
• Figure 4: Models learn quiet features before the phase transition within single training runs. The loss is averaged over the first third of token positions (Beginning), second third (Middle), and last third (End). Plots show compute-optimal training runs for the smallest compute budget where test accuracy is 100%. The red vertical line indicates the task success threshold, which is the training step at which the task loss starts to decrease. Horizontal dotted lines represent random baselines.
• Figure 5: Models learn different set of features (loud features) at or after the phase transition. The loss is averaged over the first third of token positions (Beginning), second third (Middle), and last third (End). The red vertical line indicates the task success threshold, which is the smallest compute budget at which the task loss starts to decrease (see Technical Appendix Figure 6). Cross-entropy loss is used for training probes for first_operand, adjacency_list. The probing loss is mean squared error for start_time and max_ending_here, since these are continuous features.