Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization

Dmitry Kovalev

TL;DR

This work builds a theoretical foundation for optimization with gradient orthogonalization by reframing orthogonalized gradient updates as non-Euclidean trust-region methods using matrix spectral and related norms. It introduces a stochastic non-Euclidean trust-region gradient method with momentum that unifies Muon, normalized SGD, and signSGD, and it provides convergence guarantees across non-convex, star-convex, and second-order smooth regimes. The paper also analyzes weight decay and extrapolation variants, yielding improved iteration complexities (notably $1/ ext{ε}^3$ for star-convex and $1/ ext{ε}^{3.5}$ with extrapolation in certain settings) and offers explanations for Muon’s practical advantages over Orthogonal-SGDM and the role of weight decay in large language-model training. Together, these results illuminate when and why gradient orthogonalization helps in deep learning and connect theory with observed empirical benefits.

Abstract

Optimization with matrix gradient orthogonalization has recently demonstrated impressive results in the training of deep neural networks (Jordan et al., 2024; Liu et al., 2025). In this paper, we provide a theoretical analysis of this approach. In particular, we show that the orthogonalized gradient method can be seen as a first-order trust-region optimization method, where the trust-region is defined in terms of the matrix spectral norm. Motivated by this observation, we develop the stochastic non-Euclidean trust-region gradient method with momentum, which recovers the Muon optimizer (Jordan et al., 2024) as a special case, along with normalized SGD and signSGD with momentum (Cutkosky and Mehta, 2020; Sun et al., 2023). In addition, we prove state-of-the-art convergence results for the proposed algorithm in a range of scenarios, which involve arbitrary non-Euclidean norms, constrained and composite problems, and non-convex, star-convex, first- and second-order smooth functions. Finally, our theoretical findings provide an explanation for several practical observations, including the practical superiority of Muon compared to the Orthogonal-SGDM algorithm of Tuddenham et al. (2022) and the importance of weight decay in the training of large-scale language models.

Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization

TL;DR

This work builds a theoretical foundation for optimization with gradient orthogonalization by reframing orthogonalized gradient updates as non-Euclidean trust-region methods using matrix spectral and related norms. It introduces a stochastic non-Euclidean trust-region gradient method with momentum that unifies Muon, normalized SGD, and signSGD, and it provides convergence guarantees across non-convex, star-convex, and second-order smooth regimes. The paper also analyzes weight decay and extrapolation variants, yielding improved iteration complexities (notably for star-convex and with extrapolation in certain settings) and offers explanations for Muon’s practical advantages over Orthogonal-SGDM and the role of weight decay in large language-model training. Together, these results illuminate when and why gradient orthogonalization helps in deep learning and connect theory with observed empirical benefits.

Abstract

Optimization with matrix gradient orthogonalization has recently demonstrated impressive results in the training of deep neural networks (Jordan et al., 2024; Liu et al., 2025). In this paper, we provide a theoretical analysis of this approach. In particular, we show that the orthogonalized gradient method can be seen as a first-order trust-region optimization method, where the trust-region is defined in terms of the matrix spectral norm. Motivated by this observation, we develop the stochastic non-Euclidean trust-region gradient method with momentum, which recovers the Muon optimizer (Jordan et al., 2024) as a special case, along with normalized SGD and signSGD with momentum (Cutkosky and Mehta, 2020; Sun et al., 2023). In addition, we prove state-of-the-art convergence results for the proposed algorithm in a range of scenarios, which involve arbitrary non-Euclidean norms, constrained and composite problems, and non-convex, star-convex, first- and second-order smooth functions. Finally, our theoretical findings provide an explanation for several practical observations, including the practical superiority of Muon compared to the Orthogonal-SGDM algorithm of Tuddenham et al. (2022) and the importance of weight decay in the training of large-scale language models.

Paper Structure

This paper contains 39 sections, 25 theorems, 94 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Main Motivation: Optimization with Orthogonalized Gradients
  3. Summary of Contributions and Related Work
  4. Non-Euclidean Trust-Region Gradient Method
  5. Gradient Descent as Majorization-Minimization
  6. Non-Euclidean Trust-Region Gradient Method
  7. Gradient Orthogonalization for Matrix Function Optimization
  8. Convergence Analysis for Non-Convex Functions
  9. Stochastic Non-Euclidean Trust-Region Gradient Method with Momentum
  10. The Algorithm
  11. Convergence Analysis for Non-Convex Functions
  12. Algorithms with Weight Decay for Star-Convex Functions
  13. Non-Euclidean Trust-Region Gradient Methods with Weight Decay
  14. Convergence Analysis for Star-Convex Functions
  15. Algorithms with Extrapolation for Second-Order Smooth Functions
  16. ...and 24 more sections

Key Result

Theorem 1

Let eq:L hold and let $x_0 \in \mathop{\mathrm{dom}}\nolimits R$. Then the iterations eq:tr satisfy the following inequality: where $\hat{\nabla} R_k \in \partial R(x_k)$, $\Delta_0 = F(x_0) - \inf_x F(x)$.

Theorems & Definitions (25)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Corollary 2
  • Theorem 3
  • Theorem 4
  • Corollary 3
  • Corollary 4
  • Theorem 5
  • Theorem 6
  • ...and 15 more