Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization
Shuntaro Nagashima, Hideaki Iiduka
TL;DR
This paper investigates the convergence of the Muon optimizer, an orthogonalized first-order method for nonconvex optimization. Through a direct, simplified analysis that avoids restrictive update-rule assumptions, it establishes sharper convergence guarantees (Theorem 1) and derives concrete rates under practical learning-rate and batch-size schemes, without relying on the Polyak–Łojasiewicz condition. The results show that appropriate scheduling—such as a diminishing learning rate combined with exponentially growing batch sizes—can achieve fast rates like $O(\tfrac{1}{T})$ or $O(\tfrac{\log T}{\sqrt{T}})$ depending on the schedule, outperforming prior bounds. The findings offer theoretical insight and actionable guidance for hyperparameter tuning in Muon-based training of large-scale nonconvex models, with implications for a broader class of orthogonalized first-order methods.
Abstract
The Muon optimizer has recently attracted attention due to its orthogonalized first-order updates, and a deeper theoretical understanding of its convergence behavior is essential for guiding practical applications; however, existing convergence guarantees are either coarse or obtained under restrictive analytical settings. In this work, we establish sharper convergence guarantees for the Muon optimizer through a direct and simplified analysis that does not rely on restrictive assumptions on the update rule. Our results improve upon existing bounds by achieving faster convergence rates while covering a broader class of problem settings. These findings provide a more accurate theoretical characterization of Muon and offer insights applicable to a broader class of orthogonalized first-order methods.