Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What is a Sketch-and-Precondition Derivation for Low-Rank Approximation? Inverse Power Error or Inverse Power Estimation?

Ruihan Xu, Yiping Lu

TL;DR

This work addresses the challenge of efficient randomized low-rank approximation by reframing sketching as a preconditioning tool rather than a mere dimensionality reduction. The authors introduce Error-Powered Sketched Inverse Iteration (EPSI), which applies sketched inverse iteration to the sketching error, ensuring the true eigenvector remains a fixed point and enabling convergence that improves with the sketch size. They extend EPSI to compute the top k singular vectors via Lazy-EPSI, using orthogonalization and Nyström-based inverses to achieve linear-quadratic convergence with rates that depend on spectral gaps rather than all intermediate gaps. The Newton-Sketch interpretation clarifies the connection to second-order methods while avoiding the need for exact Hessian solves, and numerical experiments on dense and sparse matrices confirm favorable runtimes and stability against established methods like Davidson’s and Inexact RQI. Overall, EPSI provides a flexible, provably convergent, scalable framework for randomized eigenvalue and k-SVD computations with practical benefits for large-scale spectral analysis.

Abstract

Randomized sketching accelerates large-scale numerical linear algebra by reducing computational complexity. While the traditional sketch-and-solve approach reduces the problem size directly through sketching, the sketch-and-precondition method leverages sketching to construct a computational friendly preconditioner. This preconditioner improves the convergence speed of iterative solvers applied to the original problem, maintaining accuracy in the full space. Furthermore, the convergence rate of the solver improves at least linearly with the sketch size. Despite its potential, developing a sketch-and-precondition framework for randomized algorithms in low-rank matrix approximation remains an open challenge. We introduce the Error-Powered Sketched Inverse Iteration (EPSI) Method via run sketched Newton iteration for the Lagrange form as a sketch-and-precondition variant for randomized low-rank approximation. Our method achieves theoretical guarantees, including a convergence rate that improves at least linearly with the sketch size.

What is a Sketch-and-Precondition Derivation for Low-Rank Approximation? Inverse Power Error or Inverse Power Estimation?

TL;DR

This work addresses the challenge of efficient randomized low-rank approximation by reframing sketching as a preconditioning tool rather than a mere dimensionality reduction. The authors introduce Error-Powered Sketched Inverse Iteration (EPSI), which applies sketched inverse iteration to the sketching error, ensuring the true eigenvector remains a fixed point and enabling convergence that improves with the sketch size. They extend EPSI to compute the top k singular vectors via Lazy-EPSI, using orthogonalization and Nyström-based inverses to achieve linear-quadratic convergence with rates that depend on spectral gaps rather than all intermediate gaps. The Newton-Sketch interpretation clarifies the connection to second-order methods while avoiding the need for exact Hessian solves, and numerical experiments on dense and sparse matrices confirm favorable runtimes and stability against established methods like Davidson’s and Inexact RQI. Overall, EPSI provides a flexible, provably convergent, scalable framework for randomized eigenvalue and k-SVD computations with practical benefits for large-scale spectral analysis.

Abstract

Randomized sketching accelerates large-scale numerical linear algebra by reducing computational complexity. While the traditional sketch-and-solve approach reduces the problem size directly through sketching, the sketch-and-precondition method leverages sketching to construct a computational friendly preconditioner. This preconditioner improves the convergence speed of iterative solvers applied to the original problem, maintaining accuracy in the full space. Furthermore, the convergence rate of the solver improves at least linearly with the sketch size. Despite its potential, developing a sketch-and-precondition framework for randomized algorithms in low-rank matrix approximation remains an open challenge. We introduce the Error-Powered Sketched Inverse Iteration (EPSI) Method via run sketched Newton iteration for the Lagrange form as a sketch-and-precondition variant for randomized low-rank approximation. Our method achieves theoretical guarantees, including a convergence rate that improves at least linearly with the sketch size.

Paper Structure

This paper contains 27 sections, 5 theorems, 66 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contribution
  3. Preliminary
  4. Power and Inverse Power Methods
  5. Nyström Approximation
  6. Preconditioning and (Jacobi-)Davidson’s Method for Eigenvalue Computation
  7. Relationship with Inexact Rayleigh Quotient-Type Methods
  8. Notation
  9. Error-Powered Sketched Inverse Iteration
  10. Convergence Analysis of Error-Powered Sketched Inverse Iteration
  11. EPSI for Computing $k$-SVD
  12. Lazy-EPSI
  13. Fast Computation of Step 6 of Algorithm \ref{['alg:iterative_matrix_approximation']} via Nyström Approximation
  14. Convergence Rate of Lazy-EPSI
  15. Orthogonalization step improves $\|(\lambda_iI-\Lambda_1)V_1^\top u^i_{q}\|$ Quadratically.
  16. ...and 12 more sections

Key Result

Lemma 1

If $u^\ast$ is an unit eigenvector of the matrix $A$, i.e.$Au^\ast=\lambda^\ast u^\ast$, then $u^\ast$ is a fixed point of Error-Powered Sketched Inverse Iteration.

Figures (6)

  • Figure 1: Lazy-EPSI on Synthetic Matrix.
  • Figure 2: Comparison of Lazy-EPSI and Subspace Iteration on synthetic squared matrix with respect to the error of rayleigh quotient eigenvalue estimation $|\hat{\lambda}_i-\lambda_i|$.
  • Figure 3: Comparison of EPSI with k-dim Nystrom approximation and Davidson's method on synthetic $2000\times 2000$ matrix with respect to the convergence of component on the last $n-1$ eigenvectors $\|V_2^\top \hat{u}_i\|$.
  • Figure 4: Comparison of EPSI with k-dim Nystrom approximation and Davidson's method on synthetic $2000\times 2000$ matrix with respect to the convergence of component on last $n-1$ eigenvectors $\|V_2^\top \hat{u}_i\|$.
  • Figure 5: Comparison of EPSI and inexact RQI on low rank matrix (left) and dense matrix (right). The shadow area is the 0.1 quantile to 0.9 quantile of the convergence path in 50 independent experiments.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • proof
  • Lemma 5: allen2016lazysvd Lemma B.2
  • proof