Convergence of Muon with Newton-Schulz

TL;DR

This work analyzes Muon, an optimizer for matrix-structured parameters, when the orthogonalization step is implemented via a small number of Newton–Schulz iterations instead of an exact SVD. It proves nonconvex convergence to an $oldsymbol{ extepsilon}$-stationary point with rates matching the SVD-polar benchmark up to a multiplicative factor $oldsymbol{ ext{χ}}_q$ that decays doubly exponentially in the NS step count $q$, and shows the polar approximation error $oldsymbol{ extvarepsilon}_q$ also decays doubly exponentially, improving with the degree $oldsymbol{ ext{kappa}}$ of the NS polynomial. The results imply that a few NS steps suffice to achieve near-exact-polar behavior while dramatically reducing per-iteration cost, yielding faster wall-clock convergence than SVD-based approaches and sharpening rank-dependence relative to SGD with momentum. By comparing with SGD and Muon with SVD, the paper explains Muon’s practical performance and provides a principled justification for the NS-design choice. Overall, the theory narrows the practice–theory gap for Muon and reinforces the benefits of SVD-free, matrix-aware optimization in deep learning.

Abstract

We analyze Muon as originally proposed and used in practice -- using the momentum orthogonalization with a few Newton-Schulz steps. The prior theoretical results replace this key step in Muon with an exact SVD-based polar factor. We prove that Muon with Newton-Schulz converges to a stationary point at the same rate as the SVD-polar idealization, up to a constant factor for a given number $q$ of Newton-Schulz steps. We further analyze this constant factor and prove that it converges to 1 doubly exponentially in $q$ and improves with the degree of the polynomial used in Newton-Schulz for approximating the orthogonalization direction. We also prove that Muon removes the typical square-root-of-rank loss compared to its vector-based counterpart, SGD with momentum. Our results explain why Muon with a few low-degree Newton-Schulz steps matches exact-polar (SVD) behavior at a much faster wall-clock time and explain how much momentum matrix orthogonalization via Newton-Schulz benefits over the vector-based optimizer. Overall, our theory justifies the practical Newton-Schulz design of Muon, narrowing its practice-theory gap.

Figures (11)

• Figure 1: $\textsc{Newton--Schulz}$ steps ($q$) ablation.$\textsc{Muon}$ with $\textsc{Newton--Schulz}$ for $q\in\{1,2,3\}$ vs. $\textsc{Muon}$ (SVD) and $\textsc{SGD}$ with momentum (SGD‑M, baseline).
• Figure 2: Train losses of MLP on MNIST across wall-clock time and epochs
• Figure 3: Train losses of CifarNet on CIFAR-10 across wall-clock time and epochs
• Figure 4: Train losses of ResNet-18 on CIFAR-100 across wall-clock time and epochs
• Figure 5: Train losses of WideResNet-28-10 on Tiny-ImageNet across wall-clock time and epochs

Theorems & Definitions (46)

• Definition 1: $\epsilon$-stationary point
• Definition 2: $\textsc{Newton--Schulz}$ polynomial
• Proposition 1: Properties of $p_\kappa$
• Definition 3: Orthogonality residual and polar approximation error
• Remark 1
• Theorem 1: Convergence of $\textsc{Muon}$ with $\textsc{Newton--Schulz}$
• Theorem 2: Upper-bounds on $\varepsilon_q$ and $\chi_q$
• Lemma 1: Orthogonality residual vs. Polar approximation error
• Lemma 2: Residual update
• Lemma 3: Residual decay by $\textsc{Newton--Schulz}$ polynomial