Riemannian Lyapunov Optimizer: A Unified Framework for Optimization

TL;DR

This work presents a unified geometric framework for optimization by reinterpreting training as a controlled dynamical system on a Riemannian manifold and introducing the normally attracting invariant manifold (NAIM) as the organizing scaffold. A strict Lyapunov function certifies convergence to a target velocity graph, enabling a constructive Lyapunov-based design that yields the Riemannian Lyapunov Optimizer (RLO) family. By mapping existing optimizers to geometric components (the metric $g$, the direction field $\Phi$, and the lifting parameter $\eta$), the authors recover SGD with momentum, AdamW, and Lion as special cases while providing principled two-time-scale dynamics and robustness guarantees under stochastic gradients. Large-scale experiments on ImageNet demonstrate that RLO variants, particularly RLO-$\Lambda$, achieve state-of-the-art or competitive accuracy across CNNs and vision transformers, highlighting the framework’s practical impact for stable, high-performance optimization. Overall, the NAIM-Lyapunov perspective bridges control theory and deep learning optimization, offering a systematic toolkit for designing stable, effective optimizers with principled convergence guarantees.

Abstract

We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework. Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. This perspective yields a constructive ``optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers.

Figures (4)

• Figure 1: Geometric intuition of the NAIM Lyapunov framework. The orange surface represents the NAIM embedded in the extended state space $\theta, v$. Blue streamlines illustrate the fast dynamics: the velocity state $v$ rapidly contracts onto $\Lambda$ at a rate governed by the lifting parameter $\eta$, regardless of initial conditions. Orange arrows show the slow dynamics: once the trajectory reaches $\Lambda$ , the system evolves along the direction field $\Phi$ toward the target optimum at the center.
• Figure 2: Hyperparameter sensitivity heatmaps for the four factorial variants. Color intensity indicates test accuracy (%). The global normalization enabled variants (left column) achieve peak performance at small learning rates ($\sim 3 \times 10^{-5}$), while the without global normalization variants (right column) require large learning rates ($\sim 3 \times 10^{-3}$). The optimal regions differ by approximately two orders of magnitude in the learning rate axis.
• Figure 3: Validation accuracy curves for ImageNet classification using ResNet-50 (left), ViT-B/16 (center), and ViT-S/16 (right). RLO-$\Lambda$ consistently achieves the highest final accuracy across all architectures (73.98%, 76.47%, 76.18% respectively). Notably, sign-based optimizers (Lion, RLO variants) demonstrate substantially faster convergence and higher final accuracy than AdamW on Vision Transformers, suggesting that the NAIM-guided updates are particularly effective for attention-based architectures.
• Figure 6: $\eta$ ablation test. (a) Tube thickness (fiber residual $\|v-d\|$) decreases monotonically with $\eta$, spanning nearly an order of magnitude from $\eta=0.1$ ($\sim$ 4000) to $\eta=0.9$ ($\sim$ 400). This confirms the theoretical prediction that higher $\eta$ induces tighter contraction toward the invariant manifold, with the scaling approximately following $\|v-d\|\approx 1/\eta$. (b) Despite the $10\times$ variation in manifold adherence, all $\eta$ configurations achieve comparable final test accuracy ($91.2\%-92.0\%$), demonstrating the robustness of the NAIM framework. Notably, $\eta=0.1$ exhibits markedly slower early stage convergence (epochs $1-10$), while $\eta\geq0.3$ configurations show nearly identical learning dynamics. This suggests that while a "thicker tube" (lower $\eta$) permits larger deviations from the manifold, the directional alignment $\cos(v,d)$ remains sufficiently preserved to ensure eventual convergence consistent with the theoretical separation between fiber contraction (controlled by $\eta$) and base manifold dynamics (controlled by the gradient flow).

Theorems & Definitions (6)

• Theorem 3.1
• Proposition B.1
• proof
• Definition C.1: Riemannian $L$-Smoothness
• Remark C.4
• Definition C.5: Polyak-Lojasiewicz Condition.