Fantastic Pretraining Optimizers and Where to Find Them

TL;DR

This work challenges the prevailing view that alternative optimizers dramatically accelerate language model pretraining by showing that fair, end-of-training benchmarking with optimizer-specific hyperparameter tuning yields much smaller speedups than previously claimed. The authors implement a three-phase methodology—Phase I for fine-grained hyperparameter tuning, Phase II for scaling-sensitive parameter tuning, and Phase III for hyperparameter scaling laws—to compare 11 optimizers across four model sizes and a wide range of data budgets. They find matrix-based optimizers (e.g., Muon, Soap, Kron) outperform scalar ones at small scales, but their relative advantage diminishes with model size, culminating in roughly 1.1× gains for 1.2B models; no method achieves the touted 2× improvement. The paper emphasizes rigorous benchmarking, reveals the sensitivity of optimizer rankings to data budgets and hyperparameters, and discusses implications for designing optimizers that retain speedups as models scale.

Abstract

AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.

Figures (8)

• Figure 1: Top Left: The commonly used AdamW baseline for optimizer design is under-tuned. Up to a 2$\times$ speedup is achievable by tuning a single hyperparameter (learning rate) in the GPT-3 recipe brown2020languagemodelsfewshotlearners for a 100M model (adopted in liu2024sophiascalablestochasticsecondorderwen2024understandingwarmupstabledecaylearningratesyuan2025marsliang2025cautiouswang2025sharpnessdisparityprincipletransformers), highlighting the importance of proper hyperparameter optimization. Top Right: Fixing hyperparameters across optimizers does not guarantee fair comparison. Shared hyperparameters such as learning rate and weight decay are commonly set to a constant in previous studies. However, even conceptually similar optimizers may correspond to very different optimal hyperparameters. Bottom Left: Speedup decays with model size. While some optimizers show high (1.3-1.4$\times$) speedup over AdamW on models under 1B parameters, the speedup decays with model size to only 1.1$\times$ for 1.2B parameters. Bottom Right: Matrix-based optimizers consistently outperform scalar-based optimizers. The loss curves for three scalar-based optimizers (AdamW, Nesterov AdamW, Mars) and three matrix-based optimizers (Kron, Soap, Muon) trained with different Chinchilla ratios of data are shown. Matrix-based optimizers achieve a consistent speedup over scalar-based optimizers. Furthermore, the three matrix-based optimizers converge to a similar loss in an overtrained setting.
• Figure 2: Main Results For Phase I & II. Top: We plot the validation loss on C4/EN for the experiments in Phase I and Phase II. Every point corresponds to the optimal loss achieved at the corresponding Chinchilla ratio for each optimizer. Bottom: we plot the HellaSwag performance corresponding to the selected run for a subset of optimizers: the AdamW baseline, the top 2 most performant scalar-based optimizers, and the top 3 most performant matrix-based optimizers. Analysis is deferred to \ref{['sec:opt-comp-gen']}.
• Figure 3: Speedup of different optimizers across scale. We estimate the speedup of different optimizers by fitting a scaling law for AdamW and then map the loss of different optimizers to the corresponding equivalent data budget. We observe that (i) The highest speedup is capped at 1.4$\times$; (ii) matrix-based optimizers consistently outperform scalar-based optimizers and show an increasing speedup with data budget.
• Figure 4: Case Studies. Left: Validation loss scaling on 1.2B model for AdamW, NAdamW, Muon and Soap. Muon and Soap still offer significant speedup over AdamW but no longer significantly speed up over NAdamW. Mid: Estimated speedup ratio with the same methodology \ref{['fig:speedup']}, we observe that Muon and Soap's speedup decays with model size to only $1.1\times$. Last: Experiment with 300M $16 \times$ Chinchilla setting, Soap outperforms Muon when data-to-model ratio further increases.
• Figure 5: Necessity of Careful Tuning. Left: 2$\times$ the optimal learning rate diminishes Soap's loss improvement over Mars on 520M 8x experiment; Mid: Variation of loss when only one hyperparameter differs from optimal learning rate and runs converge to within $0.02$ of optimal. The order of optimizers may flip arbitrarily if rigorous tuning is missing. Right: Changing a single hyperparameter like weight decay may lead to misleading faster loss improvement but plateaus later.