Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entrywise error bounds for low-rank approximations of kernel matrices

Alexander Modell

Abstract

In this paper, we derive entrywise error bounds for low-rank approximations of kernel matrices obtained using the truncated eigen-decomposition (or singular value decomposition). While this approximation is well-known to be optimal with respect to the spectral and Frobenius norm error, little is known about the statistical behaviour of individual entries. Our error bounds fill this gap. A key technical innovation is a delocalisation result for the eigenvectors of the kernel matrix corresponding to small eigenvalues, which takes inspiration from the field of Random Matrix Theory. Finally, we validate our theory with an empirical study of a collection of synthetic and real-world datasets.

Entrywise error bounds for low-rank approximations of kernel matrices

Abstract

In this paper, we derive entrywise error bounds for low-rank approximations of kernel matrices obtained using the truncated eigen-decomposition (or singular value decomposition). While this approximation is well-known to be optimal with respect to the spectral and Frobenius norm error, little is known about the statistical behaviour of individual entries. Our error bounds fill this gap. A key technical innovation is a delocalisation result for the eigenvectors of the kernel matrix corresponding to small eigenvalues, which takes inspiration from the field of Random Matrix Theory. Finally, we validate our theory with an empirical study of a collection of synthetic and real-world datasets.
Paper Structure (29 sections, 12 theorems, 118 equations, 2 figures, 1 table)

This paper contains 29 sections, 12 theorems, 118 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Big-$\mathcal{O}$ notation and frequent events
  5. Setup
  6. Entrywise error bounds
  7. Special cases
  8. Comparison with random projections and the Johnson-Lindenstrauss lemma
  9. Proof of Theorem \ref{['thm:entrywise_bound']}
  10. Experiments
  11. Datasets and setup
  12. Interpretation of the results
  13. Limitations and open problems
  14. Limitations of our theory
  15. Open problems
  16. ...and 14 more sections

Key Result

Theorem 1

Suppose that $k$ is a symmetric, positive-definite, continuous and bounded kernel and $\rho$ is a probability measure which satisfy (R) and one of either (P) or (E). If the hypothesis (P) holds and $d = \Omega\left( n^{1/\alpha} \right)$, then with overwhelming probability. If the hypothesis (E) holds and $d > \log^{1/\gamma}(n^{1/\beta})$, then with overwhelming probability.

Figures (2)

  • Figure 1: The maximum entrywise error against rank for low-rank approximations of kernel matrices constructed from a collection of datasets. The kernel matrices are constructed using Matérn kernels with a range of smoothness parameters, each of which is represented by a line in each plot. Details of the experiment are provided in Section \ref{['sec:experiments']}.
  • Figure 2: The Frobenius-norm error against rank for low-rank approximations of kernel matrices constructed from a collection of datasets. The kernel matrices are constructed using Matérn kernels with a range of smoothness parameters, each of which is represented by a line in each plot. Details of the experiment are provided in Section \ref{['sec:experiments']}.

Theorems & Definitions (13)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Theorem 2
  • Theorem 3
  • proof : Proof of Theorem \ref{['thm:delocalisation']}
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • ...and 3 more