Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Delving into Muon and Beyond: Deep Analysis and Extensions

Xianbiao Qi, Marco Chen, Jiaquan Ye, Yelin He, Rong Xiao

TL;DR

This work introduces a unified spectral framework for matrix-based optimizers by treating Muon as the $p=0$ endpoint of $\Psi_p(O)=U\Sigma^{p}V^{\top}$, with fractional variants $p\in\{1/2,1/4,1\}$ and both first-moment and RMS-normalized inputs. It develops a coupled Newton-Schulz method to compute fractional transforms without explicit SVD and studies eight optimizer instances derived from momentum and RMS inputs. Across a controlled GPT-2-like experiment, RMS-normalized updates prove more stable than first-moment updates, and spectral compression aids stability for first-moment updates, though Muon does not consistently outperform Adam; second-moment based methods remain strongest overall. The findings suggest Muon acts as spectral normalization rather than a universally superior optimizer, guiding when and how spectral techniques should be incorporated into practice. The work provides practical insights and release-ready code to enable broader adoption and further investigation of spectral approaches to optimization.

Abstract

The Muon optimizer has recently attracted considerable attention for its strong empirical performance and use of orthogonalized updates on matrix-shaped parameters, yet its underlying mechanisms and relationship to adaptive optimizers such as Adam remain insufficiently understood. In this work, we aim to address these questions through a unified spectral perspective. Specifically, we view Muon as the p = 0 endpoint of a family of spectral transformations of the form U \boldsymbolΣ^{p} V' , and consider additional variants with p = 1/2 , p = 1/4 , and p = 1 . These transformations are applied to both first-moment updates, as in momentum SGD, and to root-mean-square (RMS) normalized gradient updates as in Adam. To enable efficient computation, we develop a coupled Newton iteration that avoids explicit singular value decomposition. Across controlled experiments, we find that RMS-normalized updates yield more stable optimization than first-moment updates. Moreover, while spectral compression provides strong stabilization benefits under first-moment updates, the Muon update (p = 0) does not consistently outperform Adam. These results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method. Our source code will be released at https://github.com/Ocram7/BeyondMuon.

Delving into Muon and Beyond: Deep Analysis and Extensions

TL;DR

This work introduces a unified spectral framework for matrix-based optimizers by treating Muon as the endpoint of , with fractional variants and both first-moment and RMS-normalized inputs. It develops a coupled Newton-Schulz method to compute fractional transforms without explicit SVD and studies eight optimizer instances derived from momentum and RMS inputs. Across a controlled GPT-2-like experiment, RMS-normalized updates prove more stable than first-moment updates, and spectral compression aids stability for first-moment updates, though Muon does not consistently outperform Adam; second-moment based methods remain strongest overall. The findings suggest Muon acts as spectral normalization rather than a universally superior optimizer, guiding when and how spectral techniques should be incorporated into practice. The work provides practical insights and release-ready code to enable broader adoption and further investigation of spectral approaches to optimization.

Abstract

The Muon optimizer has recently attracted considerable attention for its strong empirical performance and use of orthogonalized updates on matrix-shaped parameters, yet its underlying mechanisms and relationship to adaptive optimizers such as Adam remain insufficiently understood. In this work, we aim to address these questions through a unified spectral perspective. Specifically, we view Muon as the p = 0 endpoint of a family of spectral transformations of the form U \boldsymbolΣ^{p} V' , and consider additional variants with p = 1/2 , p = 1/4 , and p = 1 . These transformations are applied to both first-moment updates, as in momentum SGD, and to root-mean-square (RMS) normalized gradient updates as in Adam. To enable efficient computation, we develop a coupled Newton iteration that avoids explicit singular value decomposition. Across controlled experiments, we find that RMS-normalized updates yield more stable optimization than first-moment updates. Moreover, while spectral compression provides strong stabilization benefits under first-moment updates, the Muon update (p = 0) does not consistently outperform Adam. These results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method. Our source code will be released at https://github.com/Ocram7/BeyondMuon.
Paper Structure (53 sections, 50 equations, 4 figures, 2 tables, 3 algorithms)

This paper contains 53 sections, 50 equations, 4 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Delving into Muon and Beyond
  4. Baselines: Adam and Muon
  5. Adam.
  6. Muon.
  7. Newton--Schulz approximation.
  8. Momentum $\;\text{vs.}\;$ RMS-Normalized Updates
  9. Where to apply spectral transforms.
  10. A Spectral Family of Transformations
  11. Instantiating the Spectral Family
  12. From a spectral operator to concrete optimizers.
  13. Naming convention.
  14. Momentum-input family ($\boldsymbol{O}_t^{\text{mom}}=\boldsymbol{M}_t$).
  15. RMS-normalized-input family ($\boldsymbol{O}_t^{\text{rms}}=\boldsymbol{M}_t\oslash\sqrt{\boldsymbol{V}_t}$).
  16. ...and 38 more sections

Figures (4)

  • Figure 1: Effect of spectral exponent $p$ on the singular spectrum (illustrative numbers). Decreasing $p$ compresses the spectrum: large singular values are damped relative to small ones, and $p=0$ maps all nonzero singular values to $1$.
  • Figure 2: Overall comparison across four optimizers (mSGDZ/Muon, mSGDQ, mSGDS and mSGD) based on the first-moment momentum $\boldsymbol{M}_t$. Each subfigure corresponds to a different optimizer.
  • Figure 3: Overall comparison across four optimizers (AdamZ, AdamS, AdamQ and Adam) based on the second-moment-normalized update $\boldsymbol{M}_t \oslash \sqrt{\boldsymbol{V}_t}$. Each subfigure corresponds to a different optimizer.
  • Figure 4: Comparison of four spectral exponents $p\in\{0,\tfrac{1}{4},\tfrac{1}{2},1\}$ applied to either the first-moment momentum $\boldsymbol{M}_t$ or the RMS-normalized update $\boldsymbol{M}_t \oslash \sqrt{\boldsymbol{V}_t}$, each shown at its tuned learning rate. Dashed curves correspond to momentum-input variants, while solid curves correspond to RMS-normalized-input variants.