Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum

Jingru Li, Yibo Fan, Huan Li

TL;DR

Two variants of Muon, Muon-NSR and Muon-VS are proposed, which apply variance-adaptive normalization to momentum before orthogonalization, which accelerate convergence and consistently achieve lower validation loss than both competitive, well-tuned AdamW and Muon baselines.

Abstract

Large Language Models (LLMs) achieve competitive performance across diverse natural language processing (NLP) tasks, yet pretraining is computationally demanding, making optimizer efficiency an important practical consideration. Muon accelerates LLM pretraining via orthogonal momentum updates that serve as a matrix analogue of the element-wise sign operator. Motivated by the recent perspective that Adam is a variance-adaptive sign update algorithm, we propose two variants of Muon, Muon-NSR and Muon-VS, which apply variance-adaptive normalization to momentum before orthogonalization. Muon-NSR applies noise-to-signal ratio (NSR) modulation, while Muon-VS performs variance-based scaling without introducing additional hyperparameters. Experiments on GPT-2 and LLaMA pretraining demonstrate that our proposed methods accelerate convergence and consistently achieve lower validation loss than both competitive, well-tuned AdamW and Muon baselines. For example, on the LLaMA-1.2B model, Muon-NSR and Muon-VS reduce the iterations required to reach the target validation loss by $1.36\times$ relative to the well-tuned Muon following the recent benchmark.

Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum

TL;DR

Two variants of Muon, Muon-NSR and Muon-VS are proposed, which apply variance-adaptive normalization to momentum before orthogonalization, which accelerate convergence and consistently achieve lower validation loss than both competitive, well-tuned AdamW and Muon baselines.

Abstract

Large Language Models (LLMs) achieve competitive performance across diverse natural language processing (NLP) tasks, yet pretraining is computationally demanding, making optimizer efficiency an important practical consideration. Muon accelerates LLM pretraining via orthogonal momentum updates that serve as a matrix analogue of the element-wise sign operator. Motivated by the recent perspective that Adam is a variance-adaptive sign update algorithm, we propose two variants of Muon, Muon-NSR and Muon-VS, which apply variance-adaptive normalization to momentum before orthogonalization. Muon-NSR applies noise-to-signal ratio (NSR) modulation, while Muon-VS performs variance-based scaling without introducing additional hyperparameters. Experiments on GPT-2 and LLaMA pretraining demonstrate that our proposed methods accelerate convergence and consistently achieve lower validation loss than both competitive, well-tuned AdamW and Muon baselines. For example, on the LLaMA-1.2B model, Muon-NSR and Muon-VS reduce the iterations required to reach the target validation loss by relative to the well-tuned Muon following the recent benchmark.
Paper Structure (48 sections, 17 equations, 5 figures, 6 tables, 2 algorithms)

This paper contains 48 sections, 17 equations, 5 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Works
  4. Motivation from the Variance-Adaptive Perspective of Adam
  5. An equivalent form of Adam with equal betas.
  6. Interpretation of Adam as variance-adaptive sign update.
  7. Muon: A Matrix Analogue of Signum.
  8. Motivation: Muon with variance-adaptive scaling.
  9. Muon-NSR and Muon-VS
  10. Momentum and Variance.
  11. Nesterov-style acceleration.
  12. Muon-NSR
  13. Muon-VS.
  14. Orthogonalization and Parameter Update
  15. Computational and Spatial Complexity
  16. ...and 33 more sections

Figures (5)

  • Figure 1: Validation loss trajectories for LLaMA-1.2B on C4-en (Suite B). The plot illustrates the convergence of Muon-NSR, Muon-VS, and well-tuned Muon following the recent benchmark wen2025fantastic over training iterations, with the model pretrained on the DCLM dataset under $1\times$ Chinchilla compute budget.
  • Figure 2: Comparison of training and validation loss curves on GPT-2 125M and GPT-2 355M under the setting of si2025adamuonadaptivemuonoptimizer. Curves are smoothed via exponential moving average (EMA, decay=0.95) for improved visual clarity.
  • Figure 3: Comparison of validation loss and next-token top-1 accuracy on LLaMA-130M and LLaMA-300M under the Suite B protocol following the benchmark wen2025fantastic.
  • Figure 4: Comparison of validation loss curves on LLaMA-130M and LLaMA-300M under the Suite B protocol following the benchmark wen2025fantastic.
  • Figure 5: Ablation study on the sequence of NSR modulation for LLaMA-130M, comparing Muon-NSR (pre-orthogonalization) and Muon-NSR-Reshuffled (post-orthogonalization) variants.