Understanding the Mechanisms of Fast Hyperparameter Transfer

TL;DR

This work develops a principled framework to understand hyperparameter transfer across scale in deep learning, formalizing weak transfer under width-based scaling and introducing the notions of fast and useful transfer. It distinguishes between the asymptotic convergence of optimal hyperparameters and the rate at which the evaluation loss converges, showing that weak transfer does not automatically imply compute-efficient transfer. A novel trajectory decomposition based on a linearized EMA loss and a top-$k$ subspace isolates width-stable versus width-sensitive components, providing a concrete mechanism for fast and useful HP transfer in practice. Empirical results across transformers (Llama, GPT-2), MLPs, and optimizers (Adam, SGD, Muon, Dion) reveal a consistent pattern: a width-invariant top-$k$ loss direction often governs HP choice, while residual tail directions gain width-dependent improvements, explaining when HP transfer succeeds or falters and guiding future optimization strategies.

Abstract

The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparameters, which can enable direct transfer of optimal HPs from small-scale grid searches to large models with minimal performance loss. To understand the principles governing such transfer strategy, we develop a general conceptual framework for reasoning about HP transfer across scale, characterizing transfer as fast when the suboptimality it induces vanishes asymptotically faster than the finite-scale performance gap. We show formally that fast transfer is equivalent to useful transfer for compute-optimal grid search, meaning that transfer is asymptotically more compute-efficient than direct tuning. While empirical work has found that the Maximal Update Parameterization ($μ$P) exhibits fast transfer when scaling model width, the mechanisms remain poorly understood. We show that this property depends critically on problem structure by presenting synthetic settings where transfer either offers provable computational advantage or fails to outperform direct tuning even under $μ$P. To explain the fast transfer observed in practice, we conjecture that decomposing the optimization trajectory reveals two contributions to loss reduction: (1) a width-stable component that determines the optimal HPs, and (2) a width-sensitive component that improves with width but weakly perturbs the HP optimum. We present empirical evidence for this hypothesis across various settings, including large language model pretraining.

Figures (35)

• Figure 1: Loss decomposition into top-$k$ and residual components in transformers with varying widths.
• Figure 2: Learning Gaussian $k$-index model using two-layer ReLU network with $\mu$P. The HP of interest is the Adam learning rate. Experiment details can be found in Appendix \ref{['app:two-layer-relu']}. The optimal HP exhibits an abrupt shift at $n=8192$.
• Figure 3: Illustration of convergent quantities in Definition \ref{['def:convergent_quantities']}; note that $c_n$ captures the efficiency of HP transfer.
• Figure 4: Optimal ridge penalty (generalization error) for RF regression to learn a single-index model.
• Figure 5: Optimal learning rate (validation loss) for two-layer ReLU network to learn the ball indicator function.

Theorems & Definitions (30)

• Definition 1: Convergent Quantities
• Proposition 1
• Definition 2
• Theorem 2: Useful transfer
• Theorem 3
• Theorem 4: Arzelà--Ascoli
• Proposition 5
• Proposition 6
• Theorem 7: Dynamical Dichotomy yang2021tensor
• Definition 3: HP Transfer