Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the randomized SVD in infinite dimensions

Daniel Kressner, David Persson, André Uschmajew

TL;DR

This work develops a rigorous infinite-dimensional extension of the randomized SVD for Hilbert--Schmidt operators by using isotropic Gaussian sketches in the target space, eliminating dependence on a covariance kernel in the domain. It proves error bounds that parallel the finite-dimensional RSVD, and shows that discretized implementations converge to the infinite-dimensional scheme in Wasserstein distance as the discretization refines. A decoupled analysis separates discretization error from RSVD error, and an adaptive truncation scheme is proposed to efficiently resolve the operator range in practice. The Nyström approximation is extended to SPSD trace-class operators in Hilbert spaces, with convergence results and practical discretization guidelines. Numerical experiments using Chebfun validate the theory and illustrate adaptive truncation and the impact of covariance choice on approximation quality.

Abstract

Randomized methods, such as the randomized SVD (singular value decomposition) and Nyström approximation, are an effective way to compute low-rank approximations of large matrices. Motivated by applications to operator learning, Boullé and Townsend (FoCM, 2023) recently proposed an infinite-dimensional extension of the randomized SVD for a Hilbert-Schmidt operator $A$ that invokes randomness through a Gaussian process with a covariance operator $K$. While the non-isotropy introduced by $K$ allows one to incorporate prior information on $A$, an unfortunate choice may lead to unfavorable performance and large constants in the error bounds. In this work, we introduce a novel infinite-dimensional extension of the randomized SVD that does not require such a choice and enjoys error bounds that match those for the finite-dimensional case. Our extension implicitly uses isotropic random vectors, reflecting a choice commonly made in the finite-dimensional case. In fact, the theoretical results of this work show how the usual randomized SVD applied to a discretization of $A$ approaches our infinite-dimensional extension as the discretization gets refined, both in terms of error bounds and the Wasserstein distance. We also present and analyze a novel extension of the Nyström approximation for self-adjoint positive semi-definite trace class operators.

On the randomized SVD in infinite dimensions

TL;DR

This work develops a rigorous infinite-dimensional extension of the randomized SVD for Hilbert--Schmidt operators by using isotropic Gaussian sketches in the target space, eliminating dependence on a covariance kernel in the domain. It proves error bounds that parallel the finite-dimensional RSVD, and shows that discretized implementations converge to the infinite-dimensional scheme in Wasserstein distance as the discretization refines. A decoupled analysis separates discretization error from RSVD error, and an adaptive truncation scheme is proposed to efficiently resolve the operator range in practice. The Nyström approximation is extended to SPSD trace-class operators in Hilbert spaces, with convergence results and practical discretization guidelines. Numerical experiments using Chebfun validate the theory and illustrate adaptive truncation and the impact of covariance choice on approximation quality.

Abstract

Randomized methods, such as the randomized SVD (singular value decomposition) and Nyström approximation, are an effective way to compute low-rank approximations of large matrices. Motivated by applications to operator learning, Boullé and Townsend (FoCM, 2023) recently proposed an infinite-dimensional extension of the randomized SVD for a Hilbert-Schmidt operator that invokes randomness through a Gaussian process with a covariance operator . While the non-isotropy introduced by allows one to incorporate prior information on , an unfortunate choice may lead to unfavorable performance and large constants in the error bounds. In this work, we introduce a novel infinite-dimensional extension of the randomized SVD that does not require such a choice and enjoys error bounds that match those for the finite-dimensional case. Our extension implicitly uses isotropic random vectors, reflecting a choice commonly made in the finite-dimensional case. In fact, the theoretical results of this work show how the usual randomized SVD applied to a discretization of approaches our infinite-dimensional extension as the discretization gets refined, both in terms of error bounds and the Wasserstein distance. We also present and analyze a novel extension of the Nyström approximation for self-adjoint positive semi-definite trace class operators.

Paper Structure

This paper contains 19 sections, 10 theorems, 88 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contribution and outline
  3. Preliminaries
  4. Hilbert--Schmidt operators and quasi-matrices
  5. Trace-class operators
  6. Random elements in Hilbert spaces
  7. Randomized SVD in Hilbert spaces
  8. Algorithm
  9. Analysis of \ref{['alg:infrsvd']}
  10. Finite-dimensional approximation
  11. Approximation error of randomized SVD
  12. An adaptive scheme for truncation
  13. Convergence of randomized SVD
  14. Nyström approximation in Hilbert spaces
  15. Algorithm
  16. ...and 4 more sections

Key Result

Lemma 1

Almost surely (with respect to the Gaussian measure defining $Y$) the approximation $\widehat{\mathcal{A}}:= Q Q^* \mathcal{A}$ returned by alg:infrsvd satisfies where $U_k = $ is the quasi-matrix containing the first $k$ left singular vectors of $\mathcal{A}$ and $\Omega_k$ is defined as above.

Figures (3)

  • Figure 1: Relative Hilbert--Schmidt norm errors for approximations of the integral operator defined by \ref{['eq:airy']} using Legendre polynomials. In the plots for $n=10, 15$ the dimension of the co-range basis $W_n$ is fixed in advance, whereas in Adaptive this dimension is chosen adaptively as outlined in \ref{['sec:adaptive']}. Idealized refers to the idealized algorithm outlined in \ref{['alg:infrsvd']}. Optimal denotes the optimal low-rank approximation error given by the truncated SVD.
  • Figure 2: Relative trace norm errors for approximations of the integral operator defined by \ref{['eq:sekernel']}. Optimal denotes the optimal low-rank approximation error given by the truncated SVD. $n$ refers to the truncation parameters from \ref{['section:finite_nystrom']}. Idealized refers to the idealized algorithm outlined in \ref{['alg:nystrom']}.
  • Figure 3: Relative Hilbert-Schmidt norm errors for approximations of the integral operator defined by \ref{['eq:pretty']}. Optimal denotes the optimal low-rank approximation error given by the truncated SVD. Discrete randomized SVD with $n = 258$ corresponds to the discrete randomized SVD discussed in \ref{['section:fdapprox']}. $\mathcal{K}_{\text{SE}}$ refers to the randomized SVD with random sketches drawn from $\mathcal{N}_{H_1}(0,\mathcal{K}_{\text{SE}})$. $\mathcal{K}_{\text{synth}}$ refers to the randomized SVD with random sketches drawn from $\mathcal{N}_{H_1}(0,\mathcal{K}_{\text{synth}})$.

Theorems & Definitions (22)

  • Remark 1
  • Lemma 1
  • proof
  • Theorem 2
  • proof
  • Lemma 3
  • proof
  • Theorem 4
  • proof
  • Remark 2
  • ...and 12 more