Tighter sparse variational Gaussian processes

TL;DR

The paper tackles scaling Gaussian processes by revisiting the standard SVGP posterior, and proposes a provably tighter variational approximation that relaxes the requirement q(f|u) = p(f|u). By introducing a diagonal variance modifier and deriving collapsed and uncollapsed bounds, the authors obtain tighter evidence lower bounds without added variational parameters in the Gaussian case and extend the formulation to minibatch and non-Gaussian likelihoods. The approach is integrated with SOLVE-GP and Bayesian GP-LVMs, yielding consistent improvements in predictive performance and log-marginal likelihood across regression, classification, and latent-variable tasks, while preserving computational costs. These enhancements enable more accurate uncertainty quantification and scalable GP inference, with practical impact for large datasets and complex GP models.

Abstract

Sparse variational Gaussian process (GP) approximations based on inducing points have become the de facto standard for scaling GPs to large datasets, owing to their theoretical elegance, computational efficiency, and ease of implementation. This paper introduces a provably tighter variational approximation by relaxing the standard assumption that the conditional approximate posterior given the inducing points must match that in the prior. The key innovation is to modify the conditional posterior to have smaller variances than that of the prior at the training points. We derive the collapsed bound for the regression case, describe how to use the proposed approximation in large data settings, and discuss its application to handle orthogonally structured inducing points and GP latent variable models. Extensive experiments on regression benchmarks, classification, and latent variable models demonstrate that the proposed approximation consistently matches or outperforms standard sparse variational GPs while maintaining the same computational cost. An implementation will be made available in all popular GP packages.

Figures (7)

• Figure 1: Left and middle: Optimisation traces for SGPR, T-SGPR, SVGP and T-SVGP on the Snelson dataset with 5 inducing points. Right: Predictive means and uncertainties. The stronger shade is for noiseless predictions.
• Figure 2: Test log-likelihood for various sparse approximations on eight regression datasets and various numbers of pseudo-points. For SOLVEGP and T-SOLVEGP, M is evenly split for ${\bm{u}}$ and ${\bm{v}}$. Higher is better. Best viewed in colour.
• Figure 3: Variational bound and test performance for various approximations trained on the kin40k dataset. Best viewed in colour.
• Figure 4: Log marginal likelihood approximations and test performance on the MNIST 10-way classification task. Best viewed in colour.
• Figure 5: Optimisation traces for variational Bayesian GPLVM on the oil flow dataset. Best viewed in colour.