FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer

TL;DR

FISMO addresses the tension between curvature-aware optimization and computational efficiency in training large-scale neural networks. By casting the update as a Fisher-information-based trust-region problem and approximating the Fisher blocks with a Kronecker product, FISMO achieves anisotropic preconditioning while preserving tractability. The method exposes a closed-form best-update in whitened coordinates and employs Gauss--Seidel preconditioner updates, EMA stabilization, and structured momentum to ensure stability and convergence, with an $ ext{O}(1/\,\sqrt{T})$ rate in stochastic nonconvex settings. Empirical results on image classification and language modeling show faster training, better final performance, and smoother generalization trajectories compared to baselines like Muon and AdamW, highlighting practical gains in real-world large-scale training scenarios.

Abstract

Training large-scale neural networks requires solving nonconvex optimization where the choice of optimizer fundamentally determines both convergence behavior and computational efficiency. While adaptive methods like Adam have long dominated practice, the recently proposed Muon optimizer achieves superior performance through orthogonalized momentum updates that enforce isotropic geometry with uniform singular values. However, this strict isotropy discards potentially valuable curvature information encoded in gradient spectra, motivating optimization methods that balance geometric structure with adaptivity. We introduce FISMO (Fisher-Structured Momentum-Orthogonalized) optimizer, which generalizes isotropic updates to incorporate anisotropic curvature information through Fisher information geometry. By reformulating the optimizer update as a trust-region problem constrained by a Kronecker-factored Fisher metric, FISMO achieves structured preconditioning that adapts to local loss landscape geometry while maintaining computational tractability. We establish convergence guarantees for FISMO in stochastic nonconvex settings, proving an $\mathcal{O}(1/\sqrt{T})$ rate for the expected squared gradient norm with explicit characterization of variance reduction through mini-batching. Empirical evaluation on image classification and language modeling benchmarks demonstrates that FISMO achieves superior training efficiency and final performance compared to established baselines.

Figures (3)

• Figure 1: Training and validation loss versus the number of training steps on the OpenWebText dataset (nanoGPT).
• Figure 2: Traning/validation loss/accuracy versus the number of steps on the CIFAR-10 dataset.
• Figure 3: Average condition number of the update matrices over training iterations. FISMO is compared with Adam and Muon (with 5 and 7 Newton-Schulz iterations), alongside the ideal isotropic Muon (strictly orthogonalized update).

Theorems & Definitions (28)

• Theorem 4.1: Optimal Kronecker Approximation
• Theorem 4.2
• Remark 4.3
• Remark 5.1
• Lemma 5.7
• Theorem 5.8: Main convergence theorem
• Lemma 1.1: Generalized von Neumann Trace Inequality (rectangular form)
• Theorem 1.2: Hölder inequality for Schatten norm
• Theorem 1.3: Generalized Hölder inequality for Schatten norm
• Lemma 1.4: Equivalence of Frobenius and Nuclear norm