Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Matrix perturbation analysis of methods for extracting singular values from approximate singular subspaces

Lorenzo Lazzarino, Hussam Al Daas, Yuji Nakatsukasa

TL;DR

The paper addresses how to quantify the accuracy of leading singular values retrieved from approximate left and right singular subspaces by recasting generalized Nyström as a structured matrix perturbation of the original matrix. It develops a sharp 2x2 block perturbation theory for singular values, extends it to block matrices, and provides an a posteriori computable version of the bounds. These results are applied to generalized Nyström and related extraction methods, enabling direct comparisons among GN, Rayleigh-Ritz, SVD, and HMT, with extensive numerical illustrations under both exponential and algebraic singular value decays. The findings offer practical guidance for selecting among singular-value extraction strategies in streaming and randomized settings and open avenues for improved computability and broader perturbation-based analyses.

Abstract

Given (orthonormal) approximations $\tilde{U}$ and $\tilde{V}$ to the left and right subspaces spanned by the leading singular vectors of a matrix $A$, we discuss methods to approximate the leading singular values of $A$ and study their accuracy. In particular, we focus our analysis on the generalized Nyström approximation, as surprisingly, it is able to obtain significantly better accuracy than classical methods, namely Rayleigh-Ritz and (one-sided) projected SVD. A key idea of the analysis is to view the methods as finding the exact singular values of a perturbation of $A$. In this context, we derive a matrix perturbation result that exploits the structure of such $2\times2$ block matrix perturbation. Furthermore, we extend it to block tridiagonal matrices. We then obtain bounds on the accuracy of the extracted singular values. This leads to sharp bounds that predict well the approximation error trends and explain the difference in the behavior of these methods. Finally, we present an approach to derive an a-posteriori version of those bounds, which are more amenable to computation in practice.

Matrix perturbation analysis of methods for extracting singular values from approximate singular subspaces

TL;DR

The paper addresses how to quantify the accuracy of leading singular values retrieved from approximate left and right singular subspaces by recasting generalized Nyström as a structured matrix perturbation of the original matrix. It develops a sharp 2x2 block perturbation theory for singular values, extends it to block matrices, and provides an a posteriori computable version of the bounds. These results are applied to generalized Nyström and related extraction methods, enabling direct comparisons among GN, Rayleigh-Ritz, SVD, and HMT, with extensive numerical illustrations under both exponential and algebraic singular value decays. The findings offer practical guidance for selecting among singular-value extraction strategies in streaming and randomized settings and open avenues for improved computability and broader perturbation-based analyses.

Abstract

Given (orthonormal) approximations and to the left and right subspaces spanned by the leading singular vectors of a matrix , we discuss methods to approximate the leading singular values of and study their accuracy. In particular, we focus our analysis on the generalized Nyström approximation, as surprisingly, it is able to obtain significantly better accuracy than classical methods, namely Rayleigh-Ritz and (one-sided) projected SVD. A key idea of the analysis is to view the methods as finding the exact singular values of a perturbation of . In this context, we derive a matrix perturbation result that exploits the structure of such block matrix perturbation. Furthermore, we extend it to block tridiagonal matrices. We then obtain bounds on the accuracy of the extracted singular values. This leads to sharp bounds that predict well the approximation error trends and explain the difference in the behavior of these methods. Finally, we present an approach to derive an a-posteriori version of those bounds, which are more amenable to computation in practice.
Paper Structure (15 sections, 4 theorems, 62 equations, 7 figures, 1 table, 4 algorithms)

This paper contains 15 sections, 4 theorems, 62 equations, 7 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Generalized Nyström and Matrix Perturbation
  4. Matrix Perturbation Results
  5. Application to Methods for Extracting Singular Values
  6. Application to Generalized Nyström
  7. Application to Other Extraction Methods
  8. Numerical Illustrations and Comparisons
  9. Equal Column Size Singular Approximate Subspace
  10. Different Column Size Approximate Singular Subspace
  11. Heuristic-based improvement
  12. Methods Comparison
  13. Comparison with bound in saibaba2019randomized
  14. Computability
  15. Discussion

Key Result

Theorem 4.1

\newlabelthm:2by20 Consider a $2\times 2$ block matrix with singular values $\{\sigma_i\}_{i=1}^{n}$ and the perturbed matrix with singular values $\{\hat{\sigma}_i\}_{i=1}^{n}$. Define, for $i=1, \dots, n$, Then, for all $i$ for which $\tau_i >0$, it holds

Figures (7)

  • Figure 1: Accuracy of single-pass extraction methods. We compare the approximation quality of the studied single-pass methods: generalized Nyström (red circles), Rayleigh-Ritz (green squares), and SVD (black dots). The matrix in this example is constructed such that its singular values decay exponentially. The V-shaped errors with generalized Nyström is likely an artifact of roundoff errors; in exact arithmetic, we expect the error to be roughly proportional to $1/\sigma_i$. Further details on the experiment are given in \ref{['sec:NumIll']}.
  • Figure 1: Singular value approximation errors of generalized Nyström without oversampling. We show the error in extracting the first $r = 200$ singular values by generalized Nyström approximation with $\ell=0$ (red dots), i.e., $|\sigma_i - \sigma_i^{\hbox{\tiny$GN$}}|$ for $i=1,\dots, 200$, Weyl's inequality \ref{['eq:Weyl']} (blue), and bound \ref{['eq:GNbound']} (green). Both exponential (a) and algebraic (b) singular values decays are considered.
  • Figure 2: Singular value approximation errors of generalized Nyström with oversampling. We show the error in extracting the first $r = 200$ singular values by generalized Nyström approximation with $\ell=1.5r$ (red dots), i.e., $|\sigma_i - \sigma_i^{\hbox{\tiny$GN$}}|$ for $i=1,\dots, 200$, Weyl's inequality \ref{['eq:Weyl']} (blue), bound \ref{['eq:GNbound']} (green), and the improved bound (magenta). Both exponential (a) and algebraic (b) singular values decays are considered.
  • Figure 3: Graphic representation of the process to obtain an heuristic-based bound improvement. The figures show the magnitude of the entries in logaritmic scale of the matrices $A,\Bar{A}$, and $\Bar{\Bar{A}}$, with $A$ generated with exponentially decaying singular values. The dotted lines in the figure corresponding to $\Bar{\Bar{A}}$ indicate the choice of the blocks sizes used in computing the heuristic-based bound.
  • Figure 4: Method Comparison for Exponentially decaying exact singular values. We show the approximation errors and relative bounds for: generalized Nyström (red), Rayleigh-Ritz (green), HMT (blue), and SVD (black). We consider both the case without (a) and with (b) oversampling.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Theorem 4.1
  • Corollary 4.2: Perturbation "on the right"
  • Corollary 5.1
  • Corollary 6.1