Sparse Techniques for Regression in Deep Gaussian Processes

TL;DR

STRIDE addresses the challenge of training flexible, non-stationary deep Gaussian processes (DGPs) on large-scale data by marrying sparse GP inducing-point methods with MCMC-based posterior sampling. The method formulates a Monte Carlo EM algorithm that alternates sampling the lower DGp layers with greedy updating of inducing points, achieving linear scaling in the number of observations while preserving reliable uncertainty quantification. Key contributions include (i) a principled MCEM framework for inducing-point selection in deep GPs, (ii) tractable approximations of lower-layer covariances via adaptive cross-approximation or fully sparse schemes, and (iii) extensive empirical validation on toy, UCI, and Fashion-MNIST data showing competitive accuracy and substantial computational savings over full GPR and standard sparse GPR. The work offers a practical, fully Bayesian approach to scalable deep GP regression with robust uncertainty estimates, enabling application to large datasets where standard DGPR is infeasible.

Abstract

Gaussian processes (GPs) have gained popularity as flexible machine learning models for regression and function approximation with an in-built method for uncertainty quantification. However, GPs suffer when the amount of training data is large or when the underlying function contains multi-scale features that are difficult to represent by a stationary kernel. To address the former, training of GPs with large-scale data is often performed through inducing point approximations, also known as sparse GP regression (GPR), where the size of the covariance matrices in GPR is reduced considerably through a greedy search on the data set. To aid the latter, deep GPs have gained traction as hierarchical models that resolve multi-scale features by combining multiple GPs. Posterior inference in deep GPs requires a sampling or, more usual, a variational approximation. Variational approximations lead to large-scale stochastic, non-convex optimisation problems and the resulting approximation tends to represent uncertainty incorrectly. In this work, we combine variational learning with MCMC to develop a particle-based expectation-maximisation method to simultaneously find inducing points within the large-scale data (variationally) and accurately train the deep GPs (sampling-based). The result is a highly efficient and accurate methodology for deep GP training on large-scale data. We test our method on standard benchmark problems.

Figures (9)

• Figure 1: Example of reconstructing a one-dimensional function using GP regression, sparse GP regression, and STRIDE, respectively.
• Figure 2: Illustration of 3-layer deep Gaussian processes with convolution (top) and composition (bottom) architectures. The input dimension is $d=2$. The deep GPs have been conditioned with respect to data from a simple piecewise-constant target.
• Figure 3: Comparison of errors for different values of $m$ and numbers of layers $L$ in the 1D example.
• Figure 4: Overview of obtained SMSE values in ten-fold cross-validation for three UCI datasets.
• Figure 5: Overview of obtained MNLL values in ten-fold cross-validation for three UCI datasets.