Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Riemannian optimization with finite-difference gradient approximations

Timothé Taminiau, Estelle Massart, Geovani Nunes Grapiglia

TL;DR

This work proposes a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting and presents numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.

Abstract

Derivative-free Riemannian optimization (DFRO) aims to minimize an objective function using only function evaluations, under the constraint that the decision variables lie on a Riemannian manifold. The rapid increase in problem dimensions over the years calls for computationally cheap DFRO algorithms, that is, algorithms requiring as few function evaluations and retractions as possible. We propose a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting. When endowed with an intrinsic finite-difference scheme, that measures variations of the objective in tangent directions using retractions, our proposed method requires $O(dε^{-2})$ function evaluations and retractions to find an $ε$-critical point, where $d$ is the manifold dimension. We then propose a variant of our method when the search space is a Riemannian submanifold of an $n$-dimensional Euclidean space. This variant relies on an extrinsic finite-difference scheme, approximating the Riemannian gradient directly in the embedding space, assuming that the objective function can be evaluated outside of the manifold. This approach leads to worst-case complexity bounds of $O(dε^{-2})$ function evaluations and $O(ε^{-2})$ retractions. We also present numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.

Riemannian optimization with finite-difference gradient approximations

TL;DR

This work proposes a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting and presents numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.

Abstract

Derivative-free Riemannian optimization (DFRO) aims to minimize an objective function using only function evaluations, under the constraint that the decision variables lie on a Riemannian manifold. The rapid increase in problem dimensions over the years calls for computationally cheap DFRO algorithms, that is, algorithms requiring as few function evaluations and retractions as possible. We propose a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting. When endowed with an intrinsic finite-difference scheme, that measures variations of the objective in tangent directions using retractions, our proposed method requires function evaluations and retractions to find an -critical point, where is the manifold dimension. We then propose a variant of our method when the search space is a Riemannian submanifold of an -dimensional Euclidean space. This variant relies on an extrinsic finite-difference scheme, approximating the Riemannian gradient directly in the embedding space, assuming that the objective function can be evaluated outside of the manifold. This approach leads to worst-case complexity bounds of function evaluations and retractions. We also present numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.
Paper Structure (14 sections, 9 theorems, 54 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 14 sections, 9 theorems, 54 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation:
  3. Preliminary material
  4. Basic tools and definitions in Riemannian Optimization
  5. Problem assumptions and auxiliary results
  6. A novel finite-difference DFRO method
  7. An extrinsic finite-difference scheme on manifolds
  8. Numerical results
  9. Experiments in the Euclidean setting
  10. Experiments on Riemannian test sets
  11. Problem I : Top singular vectors.
  12. Problem II : Dictionary learning.
  13. Problem III : Rotation synchronization.
  14. Conclusions

Key Result

Lemma 2.2

Given $\epsilon > 0$ and $x \in \mathcal{M}$ such that $\left\| \operatorname{grad} f(x) \right\|_{x} > \epsilon$, let $g_h(x)$ be an approximate Riemannian gradient at $x$, satisfying def:approximate_gradient for some constant $C_f>0$. If $h \leq \frac{\epsilon}{5C_f}$, then and

Figures (2)

  • Figure 1: Numerical experiments in the Euclidean case on the OPM test set Gratton2021.
  • Figure 2: Experiments for the top singular vectors problem (top row), the dictionary learning problem (second row) and the rotation synchronization problem (bottom row), on a set of problem instances described in \ref{['table:1']}.

Theorems & Definitions (21)

  • Definition 2.1
  • Lemma 2.2: Adapted from Lemma 2 in Grapiglia2023
  • proof
  • Lemma 2.3: Adapted from Lemma 1 in Grapiglia2023
  • proof
  • Theorem 3.1
  • proof
  • Lemma 3.2
  • proof
  • Lemma 3.3
  • ...and 11 more