Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Nesterov-style Accelerated Gradient Descent Algorithm for the Symmetric Eigenvalue Problem

Foivos Alimisis, Simon Vary, Bart Vandereycken

Abstract

We develop an accelerated gradient descent algorithm on the Grassmann manifold to compute the subspace spanned by a number of leading eigenvectors of a symmetric positive semi-definite matrix. This has a constant cost per iteration and a provable iteration complexity of $\tilde{\mathcal{O}}(1/\sqrtδ)$, where $δ$ is the spectral gap and $\tilde{\mathcal{O}}$ hides logarithmic factors. This improves over the $\tilde{\mathcal{O}}(1/δ)$ complexity achieved by subspace iteration and standard gradient descent, in cases that the spectral gap is tiny. It also matches the iteration complexity of the Lanczos method that has however a growing cost per iteration. On the theoretical part, we rely on the formulation of Riemannian accelerated gradient descent by [26] and new characterizations of the geodesic convexity of the symmetric eigenvalue problem by [8]. On the empirical part, we test our algorithm in synthetic and real matrices and compare with other popular methods.

A Nesterov-style Accelerated Gradient Descent Algorithm for the Symmetric Eigenvalue Problem

Abstract

We develop an accelerated gradient descent algorithm on the Grassmann manifold to compute the subspace spanned by a number of leading eigenvectors of a symmetric positive semi-definite matrix. This has a constant cost per iteration and a provable iteration complexity of , where is the spectral gap and hides logarithmic factors. This improves over the complexity achieved by subspace iteration and standard gradient descent, in cases that the spectral gap is tiny. It also matches the iteration complexity of the Lanczos method that has however a growing cost per iteration. On the theoretical part, we rely on the formulation of Riemannian accelerated gradient descent by [26] and new characterizations of the geodesic convexity of the symmetric eigenvalue problem by [8]. On the empirical part, we test our algorithm in synthetic and real matrices and compare with other popular methods.

Paper Structure

This paper contains 26 sections, 16 theorems, 124 equations, 5 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions and related work
  3. Contributions
  4. Accelerated optimization on manifolds
  5. Accelerated methods for the symmetric eigenvalue problem
  6. Riemannian accelerated gradient descent on Grassmann manifold
  7. Geometry of the Grassmann manifold and calculations for the block Rayleigh quotient
  8. Riemannian structure.
  9. Exponential map/Retractions.
  10. Principal angles.
  11. Curvature.
  12. Block Rayleigh quotient.
  13. Weak estimate sequence
  14. Towards an algorithm
  15. Remark
  16. ...and 11 more sections

Key Result

Lemma \oldthetheorem

Let $a,b,c,d$ be four points in a geodesically uniquely convex subset of a Riemannian manifold, with sectional curvatures in the interval $[-K,K]$ and then

Figures (5)

  • Figure 1: The FD3D matrix.
  • Figure 2: Comparison on ukerbe1 test matrix on problem nb 3 and 4.
  • Figure 3: Comparison on ACTIVSg70K test matrix on problem nb 5 and 6.
  • Figure 4: Comparison on boneS01 test matrix on problem 7 and 8.
  • Figure 5: Comparison on audikw_1 test matrix on problem nb 9 and 10.

Theorems & Definitions (26)

  • Lemma \oldthetheorem
  • Lemma \oldthetheorem
  • Proposition \oldthetheorem: Smoothness
  • Proposition \oldthetheorem
  • Proposition \oldthetheorem: Lemma 3 in alimisis2023gradient
  • Proposition \oldthetheorem
  • Proposition \oldthetheorem: Weak-strong convexity
  • Definition \oldthetheorem
  • Proposition \oldthetheorem
  • proof
  • ...and 16 more