Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extragradient Type Methods for Riemannian Variational Inequality Problems

Zihao Hu, Guanghui Wang, Xi Wang, Andre Wibisono, Jacob Abernethy, Molei Tao

TL;DR

This work develops REG and RPEG, Riemannian analogs of extragradient-type methods for monotone RVIPs, and proves non-asymptotic last-iterate convergence at $O\left(\frac{1}{\sqrt{T}}\right)$ as well as $O\left(\frac{1}{T}\right)$ average-iterate convergence for Riemannian minimax problems. The analysis hinges on carefully controlling the holonomy distortion arising from manifold curvature, enabling a Euclidean-proof-inspired approach via the performance estimation problem/alternatives. The results establish the first concrete non-asymptotic last-iterate guarantees for REG and RPEG on Riemannian manifolds and demonstrate their practicality as simple, single-loop first-order methods for constrained geodesically convex-concave problems. The work also outlines rigorous steps and lemmas relegated to appendices, and discusses open questions about accelerated last-iterate rates and curvature-dependent improvements, with potential extensions to stochastic/online settings and retractions.

Abstract

Riemannian convex optimization and minimax optimization have recently drawn considerable attention. Their appeal lies in their capacity to adeptly manage the non-convexity of the objective function as well as constraints inherent in the feasible set in the Euclidean sense. In this work, we delve into monotone Riemannian Variational Inequality Problems (RVIPs), which encompass both Riemannian convex optimization and minimax optimization as particular cases. In the context of Euclidean space, it is established that the last-iterates of both the extragradient (EG) and past extragradient (PEG) methods converge to the solution of monotone variational inequality problems at a rate of $O\left(\frac{1}{\sqrt{T}}\right)$ (Cai et al., 2022). However, analogous behavior on Riemannian manifolds remains an open question. To bridge this gap, we introduce the Riemannian extragradient (REG) and Riemannian past extragradient (RPEG) methods. We demonstrate that both exhibit $O\left(\frac{1}{\sqrt{T}}\right)$ last-iterate convergence. Additionally, we show that the average-iterate convergence of both REG and RPEG is $O\left(\frac{1}{T}\right)$, aligning with observations in the Euclidean case (Mokhtari et al., 2020). These results are enabled by judiciously addressing the holonomy effect so that additional complications in Riemannian cases can be reduced and the Euclidean proof inspired by the performance estimation problem (PEP) technique or the sum-of-squares (SOS) technique can be applied again.

Extragradient Type Methods for Riemannian Variational Inequality Problems

TL;DR

This work develops REG and RPEG, Riemannian analogs of extragradient-type methods for monotone RVIPs, and proves non-asymptotic last-iterate convergence at as well as average-iterate convergence for Riemannian minimax problems. The analysis hinges on carefully controlling the holonomy distortion arising from manifold curvature, enabling a Euclidean-proof-inspired approach via the performance estimation problem/alternatives. The results establish the first concrete non-asymptotic last-iterate guarantees for REG and RPEG on Riemannian manifolds and demonstrate their practicality as simple, single-loop first-order methods for constrained geodesically convex-concave problems. The work also outlines rigorous steps and lemmas relegated to appendices, and discusses open questions about accelerated last-iterate rates and curvature-dependent improvements, with potential extensions to stochastic/online settings and retractions.

Abstract

Riemannian convex optimization and minimax optimization have recently drawn considerable attention. Their appeal lies in their capacity to adeptly manage the non-convexity of the objective function as well as constraints inherent in the feasible set in the Euclidean sense. In this work, we delve into monotone Riemannian Variational Inequality Problems (RVIPs), which encompass both Riemannian convex optimization and minimax optimization as particular cases. In the context of Euclidean space, it is established that the last-iterates of both the extragradient (EG) and past extragradient (PEG) methods converge to the solution of monotone variational inequality problems at a rate of (Cai et al., 2022). However, analogous behavior on Riemannian manifolds remains an open question. To bridge this gap, we introduce the Riemannian extragradient (REG) and Riemannian past extragradient (RPEG) methods. We demonstrate that both exhibit last-iterate convergence. Additionally, we show that the average-iterate convergence of both REG and RPEG is , aligning with observations in the Euclidean case (Mokhtari et al., 2020). These results are enabled by judiciously addressing the holonomy effect so that additional complications in Riemannian cases can be reduced and the Euclidean proof inspired by the performance estimation problem (PEP) technique or the sum-of-squares (SOS) technique can be applied again.
Paper Structure (30 sections, 31 theorems, 200 equations, 1 figure, 1 table)

This paper contains 30 sections, 31 theorems, 200 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Riemannian Geometry
  5. RVIPs and Riemannian Minimax Optimization
  6. Assumptions
  7. Riemannian Extragradient
  8. Warm-up: the Euclidean Case
  9. Riemannian Extragradient and Convergence Rates
  10. Riemannian Past Extragradient
  11. Conclusion
  12. Omitted Proof for Section \ref{['sec:reg']}
  13. Proof of Lemma \ref{['lem:egpep']}
  14. Auxillary Lemmas on the Iterates of REG
  15. Proof of Lemma \ref{['lem:egbest']}
  16. ...and 15 more sections

Key Result

Lemma 1

cai2022finite Suppose hold, where $L^2 \eta^2\leq 1$, then $\|F(\mathbf{z}_{t+1})\|\leq \|F(\mathbf{z}_t)\|$.

Figures (1)

  • Figure 1: An illustration of the holonomy effect on a sphere.

Theorems & Definitions (61)

  • Definition 1: Riemannian Hamiltonian
  • Definition 2: Riemannian Primal-dual Gap
  • Definition 3
  • Remark 1
  • Remark 2
  • Lemma 1
  • Lemma 2
  • Corollary 1
  • Lemma 3
  • Remark 3
  • ...and 51 more