Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated training of Gaussian processes using banded square exponential covariances

Emily C. Ehrhardt, Felipe Tobar

TL;DR

This work introduces Banded Training Covariance (BTC), a sparsity-based approach for Gaussian process training in one dimension using the squared exponential kernel. By applying a cut-off operator to form a $k$-banded covariance $L_k(K)$, BTC reduces training complexity to $O(n k^2)$ while preserving a valid predictive distribution under conditions that ensure positive definiteness. The authors provide theoretical guarantees for structure preservation and predictive validity, and empirically demonstrate that BTC matches full GP performance with substantial runtime savings, outperforming standard sparse GP methods like FITC and VFE on real-world datasets. The method leverages the exponential decay of SE covariances to exploit almost-sparsity, offering a practical pathway for efficient GP training on long time-series, with noted limitations to the SE kernel and 1D settings and avenues for extension to higher dimensions.

Abstract

We propose a novel approach to computationally efficient GP training based on the observation that square-exponential (SE) covariance matrices contain several off-diagonal entries extremely close to zero. We construct a principled procedure to eliminate those entries to produce a \emph{banded}-matrix approximation to the original covariance, whose inverse and determinant can be computed at a reduced computational cost, thus contributing to an efficient approximation to the likelihood function. We provide a theoretical analysis of the proposed method to preserve the structure of the original covariance in the 1D setting with SE kernel, and validate its computational efficiency against the variational free energy approach to sparse GPs.

Accelerated training of Gaussian processes using banded square exponential covariances

TL;DR

This work introduces Banded Training Covariance (BTC), a sparsity-based approach for Gaussian process training in one dimension using the squared exponential kernel. By applying a cut-off operator to form a -banded covariance , BTC reduces training complexity to while preserving a valid predictive distribution under conditions that ensure positive definiteness. The authors provide theoretical guarantees for structure preservation and predictive validity, and empirically demonstrate that BTC matches full GP performance with substantial runtime savings, outperforming standard sparse GP methods like FITC and VFE on real-world datasets. The method leverages the exponential decay of SE covariances to exploit almost-sparsity, offering a practical pathway for efficient GP training on long time-series, with noted limitations to the SE kernel and 1D settings and avenues for extension to higher dimensions.

Abstract

We propose a novel approach to computationally efficient GP training based on the observation that square-exponential (SE) covariance matrices contain several off-diagonal entries extremely close to zero. We construct a principled procedure to eliminate those entries to produce a \emph{banded}-matrix approximation to the original covariance, whose inverse and determinant can be computed at a reduced computational cost, thus contributing to an efficient approximation to the likelihood function. We provide a theoretical analysis of the proposed method to preserve the structure of the original covariance in the 1D setting with SE kernel, and validate its computational efficiency against the variational free energy approach to sparse GPs.
Paper Structure (13 sections, 4 theorems, 24 equations, 4 figures, 1 table)

This paper contains 13 sections, 4 theorems, 24 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Gaussian process and sparse approximations
  3. Exploiting the almost sparse covariance
  4. Banded Training Covariance (BTC)
  5. Truncation of covariance matrices
  6. Related methods
  7. Theoretical results
  8. Structure preservation of the Cut-Off operator
  9. Validity of the Predictive Distribution
  10. Experiments
  11. Performance evaluation
  12. Validation of the theoretical choice of $k$
  13. Conclusions

Key Result

Lemma 3.1

(Positive definiteness of a cut-off matrix) Consider a symmetric positive definite matrix $A = (A_{ij}) \in \mathds{R}^{n \times n}$. Choose $k \leq n$ such that Then, the cut-off matrix $L_k(A)$ is positive definite.

Figures (4)

  • Figure 1: Square exponential kernel $k(\tau) = \sigma^2 \exp \left( - \tau^2/2\ell^2 \right)$, with $\sigma = \ell = 1$. Left: function plotted for $\tau = |x- y |$. Right: Gram matrix for 1000 equidistant points with $x_{i+1}-x_i = 0.01$.
  • Figure 1: True parameters and theoretical choice of $k$ for Fig. \ref{['fig:choice_k']}.
  • Figure 2: Sunspots dataset (3315 datapoints): NMSE (left) and NLPD (right) vs runtime. Approximation orders $m$ (FITC and VFE) and $k$ (BTC) were in the set $\{10,30,50,70,100,130,150,170,200\}$.
  • Figure 3: Neonatal EEG dataset (4000 datapoints): NMSE (left) and NLPD (right) vs runtime. Approximation orders $m$ (FITC and VFE) and $k$ (BTC) were in the set ${10,50,100,150,170,200,300,400}$.

Theorems & Definitions (13)

  • Definition 2.1
  • Definition 2.2
  • Remark 2.3
  • Remark 2.4
  • Lemma 3.1
  • proof
  • Theorem 3.2
  • proof
  • Corollary 3.3
  • proof
  • ...and 3 more