Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Invitation to Higher-Order Riemannian Optimization: Optimal and Implementable Methods

David Huckleberry Gutman, George Lobo

TL;DR

This work shows that nonconvex optimization on Riemannian manifolds can achieve the same optimal rates as in Euclidean spaces when using higher-order methods. It introduces the p-RAR algorithm, which regularizes p-th order pullback models on tangent spaces and achieves first- and second-order optimal rates under carefully designed smoothness notions that jointly depend on the objective and the retraction. A core theoretical advance is the covariant Faà di Bruno framework, which provides coordinate-free bounds on pullback derivatives via the Sasaki metric and pullback connections, enabling practical and provable analysis of higher-order methods. The paper also develops practical implementation pathways, including a Krylov-based subproblem framework for p=3 and demonstrations on Stiefel/Grassmann manifolds, indicating these higher-order Riemannian methods can be implemented and can outperform existing approaches in several regimes.

Abstract

This paper presents the first optimal-rate $p$-th order methods with $p\geq 1$ for finding first and second-order stationary points of non-convex smooth objective functions over Riemannian manifolds. In contrast to the geodesically convex setting, we definitively establish that the optimal oracle complexity of non-convex optimization over manifolds matches that over Euclidean space. In parallel with the complexity analysis, we introduce a general framework for systematically studying higher-order regularity on Riemannian manifolds that characterizes its joint dependence on the objective function and the chosen retraction. To the best of our knowledge, this framework constitutes the first known application in optimization of pullback connections and the Sasaki metric to the study of retraction-based pullbacks of the objective function. We provide clean derivative bounds based on a new covariant Faà di Bruno formula derived within our framework. For $p=3$, our methods are fully implementable via a new Krylov-based framework for minimizing quartically regularized cubic polynomials. This is the first Krylov method for this class of polynomials and may be of independent interest beyond Riemannian optimization.

An Invitation to Higher-Order Riemannian Optimization: Optimal and Implementable Methods

TL;DR

This work shows that nonconvex optimization on Riemannian manifolds can achieve the same optimal rates as in Euclidean spaces when using higher-order methods. It introduces the p-RAR algorithm, which regularizes p-th order pullback models on tangent spaces and achieves first- and second-order optimal rates under carefully designed smoothness notions that jointly depend on the objective and the retraction. A core theoretical advance is the covariant Faà di Bruno framework, which provides coordinate-free bounds on pullback derivatives via the Sasaki metric and pullback connections, enabling practical and provable analysis of higher-order methods. The paper also develops practical implementation pathways, including a Krylov-based subproblem framework for p=3 and demonstrations on Stiefel/Grassmann manifolds, indicating these higher-order Riemannian methods can be implemented and can outperform existing approaches in several regimes.

Abstract

This paper presents the first optimal-rate -th order methods with for finding first and second-order stationary points of non-convex smooth objective functions over Riemannian manifolds. In contrast to the geodesically convex setting, we definitively establish that the optimal oracle complexity of non-convex optimization over manifolds matches that over Euclidean space. In parallel with the complexity analysis, we introduce a general framework for systematically studying higher-order regularity on Riemannian manifolds that characterizes its joint dependence on the objective function and the chosen retraction. To the best of our knowledge, this framework constitutes the first known application in optimization of pullback connections and the Sasaki metric to the study of retraction-based pullbacks of the objective function. We provide clean derivative bounds based on a new covariant Faà di Bruno formula derived within our framework. For , our methods are fully implementable via a new Krylov-based framework for minimizing quartically regularized cubic polynomials. This is the first Krylov method for this class of polynomials and may be of independent interest beyond Riemannian optimization.
Paper Structure (29 sections, 27 theorems, 114 equations, 2 figures, 3 tables, 3 algorithms)

This paper contains 29 sections, 27 theorems, 114 equations, 2 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Notation
  2. Differentiation on Vector Spaces and Manifolds
  3. Retractions and Vector Transports
  4. Optimal Rate Theory, Part I: Algorithms
  5. First-Order Convergence Guarantees
  6. Second-Order Convergence Guarantees
  7. Optimal Rate Theory, Part II: Smoothness Constants
  8. The Covariant Faá di Bruno Formula
  9. Simpler Derivative Formulas via Double Tangent Bundle Calculus
  10. Joint Bounds on Higher-Order Smoothness Constants
  11. Comparison with Prior Analyses
  12. Toward Implementation, Part I: Practical Retractions on Two Manifolds
  13. Toward Implementation, Part II: Solving the $p=3$ Subproblem
  14. The Symmetric Maximal Krylov Iteration
  15. Krylov Hybrid Minimization Framework
  16. ...and 14 more sections

Key Result

Lemma 2.1

Let $x\in{\cal M}$, $\alpha>0$, and $f\in C^p({\cal M})$. If $v\in {\cal T}_x{\cal M}$ ensures the $p$-th order regularized model at $x$ decreases, in the sense that then $v$ ensures the sufficient decrease condition for the $p$-th order unregularized model. This holds for the $v_i$ computed in eq:alg-model-decrease-condition of $p$-RAR.

Figures (2)

  • Figure 1: Gradient norm versus $3$-RAR iteration count for representative instances on $(10,5)$, $(20,5)$, and $(50,5)$ setups.
  • Figure 2: Gradient norm versus total CPU time for representative instances on $(10,5)$, $(20,5)$, and $(50,5)$ setups.

Theorems & Definitions (55)

  • Definition : $(L,p,R)$-Majorization Smooth
  • Definition : $(L,p,R,\mathop{\mathrm{VT}}\nolimits)$-Smooth
  • Lemma 2.1: $p$-th Order Model Sufficient Decrease
  • Lemma 2.2: Majorization Smoothness Consequences
  • Definition : Short-Step Lower Bound
  • Lemma 2.3: Increment-to-Gradient Bound
  • proof
  • Theorem 2.4: Optimal First-Order Complexity Bound
  • proof
  • Definition : Second-Order Retraction
  • ...and 45 more