Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Gaussian process learning via subspace projections

Elsa Cazelles, Felipe Tobar

TL;DR

This work tackles the high cost of training Gaussian processes (GPs) on moderately large datasets and bias risks in sparse approximations. It introduces projected likelihood (PL), which trains a GP from the likelihood of a surrogate on a $k$-dimensional projection of the data, achieving a cost of $O(k n^2)$ and quantifiable information loss that decreases with random projections on the unit sphere. The authors provide a theoretical characterization of the information loss and demonstrate that PL, using fixed low-dimensional projections (including spherical projections), often matches or closely approaches the exact GP while outperforming variational sparse methods in practice. The results demonstrate PL's potential to enable accurate and efficient GP learning for time-series and related data by leveraging low-dimensional projections and interdomain-like inducing representations.

Abstract

We propose a novel training objective for GPs constructed using lower-dimensional linear projections of the data, referred to as \emph{projected likelihood} (PL). We provide a closed-form expression for the information loss related to the PL and empirically show that it can be reduced with random projections on the unit sphere. We show the superiority of the PL, in terms of accuracy and computational efficiency, over the exact GP training and the variational free energy approach to sparse GPs over different optimisers, kernels and datasets of moderately large sizes.

Efficient Gaussian process learning via subspace projections

TL;DR

This work tackles the high cost of training Gaussian processes (GPs) on moderately large datasets and bias risks in sparse approximations. It introduces projected likelihood (PL), which trains a GP from the likelihood of a surrogate on a -dimensional projection of the data, achieving a cost of and quantifiable information loss that decreases with random projections on the unit sphere. The authors provide a theoretical characterization of the information loss and demonstrate that PL, using fixed low-dimensional projections (including spherical projections), often matches or closely approaches the exact GP while outperforming variational sparse methods in practice. The results demonstrate PL's potential to enable accurate and efficient GP learning for time-series and related data by leveraging low-dimensional projections and interdomain-like inducing representations.

Abstract

We propose a novel training objective for GPs constructed using lower-dimensional linear projections of the data, referred to as \emph{projected likelihood} (PL). We provide a closed-form expression for the information loss related to the PL and empirically show that it can be reduced with random projections on the unit sphere. We show the superiority of the PL, in terms of accuracy and computational efficiency, over the exact GP training and the variational free energy approach to sparse GPs over different optimisers, kernels and datasets of moderately large sizes.
Paper Structure (15 sections, 3 theorems, 17 equations, 3 figures, 3 tables)

This paper contains 15 sections, 3 theorems, 17 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Gaussian processes for regression
  3. Training and inference
  4. Sparse approximations
  5. The projected likelihood method
  6. Projections onto the unit sphere $\mathcal{S} ^{n-1}$
  7. Information loss in the projected likelihood
  8. Interpretation and relationship to previous methods
  9. Design of projection directions
  10. Experiments
  11. E1: Different optimisers and variance assessment
  12. E2: Effect of the number of projections
  13. E3: Real-world series and different kernels
  14. Conclusions
  15. Acknowledgment

Key Result

Theorem 1

Denote $Y$ the random variable of the full GP evaluated on ${\mathbf x}$, and $Z= \boldsymbol{\Omega} ^\top Y$ its projection. The Fisher information loss related to learning $\theta$ through the PL in eq. eq:PL_NLL, instead of the exact NLL in eq. eq:nll, is quadratic in $\Sigma_{Y|Z}=\text{Cov}(Y|

Figures (3)

  • Figure 1: Eigenspectrum of $K_{\mathbf x}$ and $\Sigma_{Y|Z}$ using different types and number of projection matrices $\boldsymbol{\Omega}$.
  • Figure 2: Posterior variance of the GP learnt by ML, VFE and proposed PL methods, using the Adam and BFGS optimisers.
  • Figure 3: Performance [NLL] versus computation time [seconds] for VFE (red squares) and PL (blue triangles). The orders $m$ (VFE) and $k$ (PL) are denoted in each marker. The ML solution is denoted with a yellow star, with the achieved NLL in a dashed yellow line. The closer to the bottom-left corner, the better.

Theorems & Definitions (7)

  • Remark 1
  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Corollary 2
  • proof