Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mano: Restriking Manifold Optimization for LLM Training

Yufei Gu, Zeke Xie

TL;DR

This work revisits manifold optimization for large language model (LLM) pretraining and introduces Mano, a reformed optimizer that projects momentum onto the tangent space and constrains updates on a rotating Oblique manifold. Unlike traditional manifold methods that rely on costly retractions or heavy spectral preconditioning, Mano uses simple tangent-space projections and dimension-wise normalization to achieve a soft manifold constraint with low computational overhead. Empirical evidence on LLaMA and Qwen3 across multiple scales shows Mano consistently outperforms AdamW and Muon in perplexity and convergence speed, while reducing memory and compute costs relative to the baselines. Collectively, Mano expands the Pareto frontier for LLM optimization and invites further theoretical and empirical exploration of manifold-inspired training paradigms, particularly in high-dimensional regimes. $ $

Abstract

While large language models (LLMs) have emerged as a significant advancement in artificial intelligence, the hardware and computational costs for training LLMs are also significantly burdensome. Among the state-of-the-art optimizers, AdamW relies on diagonal curvature estimates and ignores structural properties, while Muon applies global spectral normalization at the expense of losing curvature information. In this study, we restriked manifold optimization methods for training LLMs, which may address both optimizers' limitations, while conventional manifold optimization methods have been largely overlooked due to the poor performance in large-scale model optimization. By innovatively projecting the momentum onto the tangent space of model parameters and constraining it on a rotational Oblique manifold, we propose a novel, powerful, and efficient optimizer **Mano** that is the first to bridge the performance gap between manifold optimization and modern optimizers. Extensive experiments on the LLaMA and Qwen3 models demonstrate that Mano consistently and significantly outperforms AdamW and Muon even with less memory consumption and computational complexity, respectively, suggesting an expanded Pareto frontier in terms of space and time efficiency.

Mano: Restriking Manifold Optimization for LLM Training

TL;DR

This work revisits manifold optimization for large language model (LLM) pretraining and introduces Mano, a reformed optimizer that projects momentum onto the tangent space and constrains updates on a rotating Oblique manifold. Unlike traditional manifold methods that rely on costly retractions or heavy spectral preconditioning, Mano uses simple tangent-space projections and dimension-wise normalization to achieve a soft manifold constraint with low computational overhead. Empirical evidence on LLaMA and Qwen3 across multiple scales shows Mano consistently outperforms AdamW and Muon in perplexity and convergence speed, while reducing memory and compute costs relative to the baselines. Collectively, Mano expands the Pareto frontier for LLM optimization and invites further theoretical and empirical exploration of manifold-inspired training paradigms, particularly in high-dimensional regimes.

Abstract

While large language models (LLMs) have emerged as a significant advancement in artificial intelligence, the hardware and computational costs for training LLMs are also significantly burdensome. Among the state-of-the-art optimizers, AdamW relies on diagonal curvature estimates and ignores structural properties, while Muon applies global spectral normalization at the expense of losing curvature information. In this study, we restriked manifold optimization methods for training LLMs, which may address both optimizers' limitations, while conventional manifold optimization methods have been largely overlooked due to the poor performance in large-scale model optimization. By innovatively projecting the momentum onto the tangent space of model parameters and constraining it on a rotational Oblique manifold, we propose a novel, powerful, and efficient optimizer **Mano** that is the first to bridge the performance gap between manifold optimization and modern optimizers. Extensive experiments on the LLaMA and Qwen3 models demonstrate that Mano consistently and significantly outperforms AdamW and Muon even with less memory consumption and computational complexity, respectively, suggesting an expanded Pareto frontier in terms of space and time efficiency.
Paper Structure (25 sections, 3 theorems, 24 equations, 8 figures, 5 tables, 2 algorithms)

This paper contains 25 sections, 3 theorems, 24 equations, 8 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Optimizers for LLM Pretraining
  4. Manifold Optimization in Deep Learning
  5. Preliminaries
  6. Methodology
  7. Reformed Manifold Optimization
  8. Manifold Selection and Design
  9. The Mano Optimizer
  10. Theoretical Analysis
  11. Experiments
  12. Experiment Setup
  13. Experiment Results
  14. Learning Dynamics
  15. Ablation Studies
  16. ...and 10 more sections

Key Result

Theorem 1

Assume that $f(\theta)$ is an $L$-smooth function, $f$ is lower bounded as $f(\theta) \geq f_{\inf}$, $\mathbb{E}[\xi]=0$ for gradient noise $\xi$ of sub-sampling, $\sin(\phi_t^{(j)}) \geq \gamma > 0$ for angle $\phi_t^{(j)}$ between $g_t^{(j)}$ and the parameter $\theta_t^{(j)}$ and the tangential where $C_1 = \frac{f(\theta_0) - f_{\inf}}{m^{\frac{1}{2}} \gamma C}, \; C_2 = \frac{L m^{\frac{3}{

Figures (8)

  • Figure 1: One day pretraining experiment of LLaMA-350M and -1.3B models on the Pile dataset for $10$B and $2.8$B tokens respectively. Our proposed optimizer Mano achieves $1.75\times$ and $1.38\times$ the convergence speed of Muon in terms of wall-clock time. This advantage is expected to further increase with reduced computational overhead and faster convergence speed.
  • Figure 2: LLaMA-350M and -1.3B models trained on the C4/en and Pile dataset for $10000$ steps with three different optimizers: AdamW, Muon, and Mano. Mano demonstrated a faster convergence speed than both popular optimizers with the simplest implementation and computational cost.
  • Figure 3: Qwen3-0.6B and -1.7B models trained on the Pile dataset for $10000$ steps with three different optimizers: AdamW, Muon, and Mano. The performance advantage of Mano is model-transferrable.
  • Figure 4: LLaMA-$130$M and -$350$M models trained on the Pile dataset for $10$B tokens. We demonstrated that with data scaling, Mano consistently performed better than Muon and AdamW in the ultimate convergence speed.
  • Figure 5: The average (a) Gradient norm, (b) Gradient variance, and (c) Gradient Signal-to-Noise Ratio (SNR) of LLaMA-$350$M model parameters trained on the Pile dataset. The SNR is calculated as the norm-to-variance ratio. As an indicator of internal training dynamics, Mano exhibits lower gradient variance and a higher SNR than Muon, both under the same momentum coefficient $\mu=0.95$.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Theorem 1: Convergence of Mano w/o Momentum
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • proof