Benchmarking Optimizers for Large Language Model Pretraining

TL;DR

This work delivers a comprehensive, standardized benchmark of 11 optimizers for large language model pretraining across multiple model sizes and training horizons, revealing that AdEMAMix, D-Muon, and MARS are strong contenders at scale after careful hyperparameter tuning. It demonstrates that batch size, warmup, weight decay, and learning-rate schedules substantially influence outcomes and that certain optimizers (e.g., Sophia, Signum, Lion) can underperform or destabilize under longer training. By extending the evaluation to Mixture-of-Experts (MoEs) and releasing full tooling, the study provides a practical framework for fair comparison and future optimizer development. The findings offer concrete guidance for practitioners and highlight directions for research into stable, efficient large-scale optimization.

Abstract

The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.

Figures (45)

• Figure 1: Ranking of optimizers for $\mathbf{720M}$ Llama-based models. We plot the final validation loss obtained by the best-tuned optimizers on the FineWeb dataset. We use a batch size of $1\mathbf{M}$ tokens and train multiple methods beyond and below the Chinchilla optimal duration, which is $14.4\mathbf{B}$ for model of this size. AdEMAMix and MARS are the best optimizers in this setup, with a noticable gap in performance compared to other methods. We also plot the AdamW baseline in both figures to distinguish the group of methods that consistently perform worse than AdamW from the group of optimizers that outperform it for some training durations. See § \ref{['sec:setup']} and \ref{['sec:ap_tuning']} for a detailed description of our experimental setup, including hyperparameters.
• Figure 2: Training dynamics of leading optimizers on $\mathbf{520M}$ MoE model pretraining. We use a batch size of $131\mathbf{k}$ tokens, and train models for both short runs, i.e., less than Chinchilla optimal duration, and for extended runs beyond this regime. The dashed blue lines correspond to the final validation loss of AdamW baselines trained for both $42\mathbf{k}$ and $336\mathbf{k}$ steps.
• Figure 3: Comparing optimizers for training a $\mathbf{124M}$ parameter LLM. We plot the validation loss dynamics obtained by considered optimizers. In (a), we train methods with a "small" batch size of $16\mathbf{k}$ tokens for $\{64, 128, 256, 384, 512, 1024\}\mathbf{k}$ iterations. In (b), we train methods with nearly $8\times$ larger batch size of $131\mathbf{k}$ tokens for $\{8, 16, 32, 48, 64, 128\}\mathbf{k}$ iterations. Thus, in both settings, we result in the same number of tokens models see during the training: $\{1, 2.1, 4.2, 6.3, 8.4, 16.8\}\mathbf{B}$. We observe that: (I) many methods outperform AdamW in the short runs for $1\mathbf{B}$ or $2.1\mathbf{B}$ tokens; (II) as training on more tokens, AdamW narrows the gap with SOAP and D-Muon, while AdEMAMix emerges as the best-performing method; (III) Signum, MARS, Lion, Prodigy benefit from the increased batch size.
• Figure 4: Weight decay in Muon & D-Muon. We compare two methods---basic Muonjordan2024muon, and D-Muonliu2025muonscalablellmtraining with a weight decay applied to all parameter groups. Across model sizes used in our benchmarking of dense LLMs, we observe a major improvement of D-Muon over Muon. We relate this observation to our ablation on the importance of weight decay across different optimizers, and training horizons. See \ref{['fig:muon-dmuon-val-loss']} and \ref{['sec:ap_wdablation']} for more details.
• Figure 5: Ranking of optimizers for $\mathbf{124}\textbf{M}$ models with "small" and "large" batch sizes. In both (a) and (b), we show the final validation loss for different training durations, corresponding to different numbers of tokens. Above each token number, we write the number of training iterations corresponding. In (a), we use a "small" batch size of $32\times512$ tokens. In (b), we use a larger batch size of $256\times512$ tokens.