Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)

TL;DR

This work addresses the gap between theory and practice in layerwise LMO-based optimizers for large models by introducing Gluon, a layerwise optimization framework paired with a generalized L0,L1-smoothness model. The approach yields adaptive, per-layer stepsizes and convergence guarantees for both deterministic and stochastic settings, recovering Muon/Scion as special cases. Empirical validation on NanoGPT-FineWeb and CIFAR-10 shows that the layerwise smoothness model holds during training and that theory-driven stepsizes closely match tuned values, with specialized per-layer norms outperforming Euclidean baselines. Overall, Gluon provides a rigorous, practically predictive foundation for structured, memory-efficient optimization in deep learning, bridging a long-standing theory-practice gap.

Abstract

Recent developments in deep learning optimization have brought about radically new algorithms based on the Linear Minimization Oracle (LMO) framework, such as $\sf Muon$ and $\sf Scion$. After over a decade of $\sf Adam$'s dominance, these LMO-based methods are emerging as viable replacements, offering several practical advantages such as improved memory efficiency, better hyperparameter transferability, and most importantly, superior empirical performance on large-scale tasks, including LLM training. However, a significant gap remains between their practical use and our current theoretical understanding: prior analyses (1) overlook the layer-wise LMO application of these optimizers in practice, and (2) rely on an unrealistic smoothness assumption, leading to impractically small stepsizes. To address both, we propose a new LMO-based method called $\sf Gluon$, capturing prior theoretically analyzed methods as special cases, and introduce a new refined generalized smoothness model that captures the layer-wise geometry of neural networks, matches the layer-wise practical implementation of $\sf Muon$ and $\sf Scion$, and leads to convergence guarantees with strong practical predictive power. Unlike prior results, our theoretical stepsizes closely match the fine-tuned values reported by Pethick et al. (2025). Our experiments with NanoGPT and CNN confirm that our assumption holds along the optimization trajectory, ultimately closing the gap between theory and practice.

Figures (12)

• Figure 1: Training NanoGPT on FineWeb validates our layer-wise $(L^0, L^1)$-smoothness model.
• Figure 2: Validation of \ref{['ass:generalized-smoothness']} for the 8th transformer block in NanoGPT-124M along training trajectories of unScion.
• Figure 3: (a) Validation curves for AdamW and unScion with varying $\rho_3$ values; (b) Heatmap of validation loss from the last iteration of unScion across different combinations of $\rho_2$ and $\rho_3$.
• Figure 4: Validation of \ref{['ass:generalized-smoothness']} for different groups of parameters in CNN along training trajectories of unScion with full-batch gradients.
• Figure 5: Validation of layer-wise $(L^0, L^1)$-smoothness for the group of parameters from the embedding layer of NanoGPT-124M along unScion training trajectories. The group norm is $\|\cdot\|_{(p)} = n_p \|\cdot\|_{1 \to \infty}$, with fitted values $L_p^0 \approx 0$, $L_p^1 \approx 1.3$. The same plot is shown twice with different $y$-axis limits.

Theorems & Definitions (21)

• Theorem 1
• Theorem 2
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4: Technical Lemma 10 by hubler2024parameter
• Lemma 5: Technical Lemma 11 by hubler2024parameter