Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomized Nyström approximation of non-negative self-adjoint operators

David Persson, Nicolas Boullé, Daniel Kressner

TL;DR

This work develops an infinite-dimensional Nyström approximation for non-negative self-adjoint trace-class operators, extending the finite-dimensional Nyström and recent operator-SVD frameworks to Hilbert spaces. The authors replace standard Gaussian sketches with Gaussian-process-based draws to create infinite-dimensional random features, derive structural and probabilistic bounds in operator, trace, and Hilbert–Schmidt norms, and establish a continuity argument to transfer finite-dimensional results to the infinite-dimensional setting. They also provide improved bounds for the infinite-dimensional randomized SVD and validate the approach via numerical experiments on Gaussian-process kernels and Bayesian inverse problems. The results enable efficient low-rank approximations of integral operators and offer practical tools for GP sampling and operator learning in scientific computing.

Abstract

The randomized singular value decomposition (SVD) has become a popular approach to computing cheap, yet accurate, low-rank approximations to matrices due to its efficiency and strong theoretical guarantees. Recent work by Boullé and Townsend (FoCM, 2023) presents an infinite-dimensional analog of the randomized SVD to approximate Hilbert-Schmidt operators. However, many applications involve computing low-rank approximations to symmetric positive semi-definite matrices. In this setting, it is well-established that the randomized Nyström approximation is usually preferred over the randomized SVD. This paper explores an infinite-dimensional analog of the Nyström approximation to compute low-rank approximations to non-negative self-adjoint trace-class operators. We present an analysis of the method and, along the way, improve the existing infinite-dimensional bounds for the randomized SVD. Our analysis yields bounds on the expected value and tail bounds for the Nyström approximation error in the operator, trace, and Hilbert-Schmidt norms. Numerical experiments on integral operators arising from Gaussian process sampling and Bayesian inverse problems are used to validate the proposed infinite-dimensional Nyström algorithm.

Randomized Nyström approximation of non-negative self-adjoint operators

TL;DR

This work develops an infinite-dimensional Nyström approximation for non-negative self-adjoint trace-class operators, extending the finite-dimensional Nyström and recent operator-SVD frameworks to Hilbert spaces. The authors replace standard Gaussian sketches with Gaussian-process-based draws to create infinite-dimensional random features, derive structural and probabilistic bounds in operator, trace, and Hilbert–Schmidt norms, and establish a continuity argument to transfer finite-dimensional results to the infinite-dimensional setting. They also provide improved bounds for the infinite-dimensional randomized SVD and validate the approach via numerical experiments on Gaussian-process kernels and Bayesian inverse problems. The results enable efficient low-rank approximations of integral operators and offer practical tools for GP sampling and operator learning in scientific computing.

Abstract

The randomized singular value decomposition (SVD) has become a popular approach to computing cheap, yet accurate, low-rank approximations to matrices due to its efficiency and strong theoretical guarantees. Recent work by Boullé and Townsend (FoCM, 2023) presents an infinite-dimensional analog of the randomized SVD to approximate Hilbert-Schmidt operators. However, many applications involve computing low-rank approximations to symmetric positive semi-definite matrices. In this setting, it is well-established that the randomized Nyström approximation is usually preferred over the randomized SVD. This paper explores an infinite-dimensional analog of the Nyström approximation to compute low-rank approximations to non-negative self-adjoint trace-class operators. We present an analysis of the method and, along the way, improve the existing infinite-dimensional bounds for the randomized SVD. Our analysis yields bounds on the expected value and tail bounds for the Nyström approximation error in the operator, trace, and Hilbert-Schmidt norms. Numerical experiments on integral operators arising from Gaussian process sampling and Bayesian inverse problems are used to validate the proposed infinite-dimensional Nyström algorithm.
Paper Structure (26 sections, 13 theorems, 99 equations, 5 figures, 1 algorithm)

This paper contains 26 sections, 13 theorems, 99 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Organization of the paper
  4. Nyström approximation in finite dimensions with correlated Gaussian sketches
  5. Distribution of Gaussian sketches
  6. Nyström method with correlated Gaussian sketches
  7. Analysis in the nuclear and operator norms
  8. Expectation bounds
  9. Tail bounds
  10. Analysis in the Frobenius norm
  11. Expectation bound
  12. Tail bound
  13. Nyström approximation in infinite dimensions
  14. Quasimatrices
  15. Non-negative self-adjoint trace-class operators
  16. ...and 11 more sections

Key Result

Theorem 2.1

\newlabeltheorem:expectation_nystrom0 Let $\bm A$ be an $n\times n$ SPSD matrix, $2\leq k\leq \mathop{\mathrm{rank}}\nolimits(\bm A)$ be a target rank, and $p\ge 2$ an oversampling parameter. Let $\bm \Omega$ be a random sketch matrix with the columns i.i.d. $\mathcal{N}(\bm{0},\bm{K})$, where the

Figures (5)

  • Figure 1: (a) Exact kernel defined by (\ref{['eq:pretty']}) along with convergence of the Nyström approximation for different values of $\ell$ (b). (c)-(e) Rank-$\textcolor{black}{100}$ Nyström approximations of the kernel for $\ell = 1,0.1,0.01$, respectively.
  • Figure 2: The contour plots (a),(c), and (e) show the Matérn kernels defined in (\ref{['eq:12']})-(\ref{['eq:52']}). The error plots (b) and (d) show the relative error in the trace norm when approximating $G_{1/2}$ and $G_{5/2}$, respectively, when using the squared exponential kernel in (\ref{['eq:sekernel']}) with $\ell = 1, 0.01$ and the Matérn-3/2 kernel in (\ref{['eq:32']}) as covariance kernels for the random fields. Optimal denotes the best low-rank approximation error. The plots in (f) show sample paths of the different Gaussian processes. The paths have been normalized so that the maximum absolute value is equal to 1.
  • Figure 3: Figure comparing the numerical results against the theoretical upper bound in (\ref{['eq:inf_expectation_tr']}) when approximating the Matérn-5/2 kernel using the squared-exponential with length parameters $\ell = 1$ and $\ell = 0.01$ and the Matérn-3/2 kernel as covariance kernels. Optimal denotes the best low-rank approximation error. The full line shows the numerical results. The dashed lines shows the theoretical upper bound in (\ref{['eq:inf_expectation_tr']}).
  • Figure 4: The left contour plot shows an exact sample from $\mathcal{GP}(0,G)$ and the left contour plot shows a sample from $\mathcal{GP}(0,G_{30})$ so that the mean-squared error equals $\|G-G_{30}\|_{\mathrm{Tr}}$. The error plots show the relative error in the trace norm. Optimal denotes the best low-rank approximation error.
  • Figure 5: The contour plots shows the solution of (\ref{['eq:pde']}) for different times. The error plots show the relative error in the trace norm norm. Optimal denotes the best low-rank approximation error.

Theorems & Definitions (28)

  • Remark 1.1
  • Theorem 2.1: Expectation bound in spectral and nuclear norms
  • Proof 1
  • Lemma 2.2
  • Proof 2
  • Theorem 2.3: Tail bound in spectral and nuclear norms
  • Proof 3
  • Remark 2.4: Connection with the randomized SVD
  • Theorem 2.5: Frobenius norm
  • Proof 4
  • ...and 18 more