Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Nearest Neighbor Gaussian Process

Luhuan Wu, Geoff Pleiss, John Cunningham

TL;DR

This work proposes variational nearest neighbor Gaussian process (VNNGP), which introduces a prior that only retains correlations within nearest-neighboring observations, thereby inducing sparse precision structure and comparing VNNGP to other scalable GPs.

Abstract

Variational approximations to Gaussian processes (GPs) typically use a small set of inducing points to form a low-rank approximation to the covariance matrix. In this work, we instead exploit a sparse approximation of the precision matrix. We propose variational nearest neighbor Gaussian process (VNNGP), which introduces a prior that only retains correlations within $K$ nearest-neighboring observations, thereby inducing sparse precision structure. Using the variational framework, VNNGP's objective can be factorized over both observations and inducing points, enabling stochastic optimization with a time complexity of $O(K^3)$. Hence, we can arbitrarily scale the inducing point size, even to the point of putting inducing points at every observed location. We compare VNNGP to other scalable GPs through various experiments, and demonstrate that VNNGP (1) can dramatically outperform low-rank methods, and (2) is less prone to overfitting than other nearest neighbor methods.

Variational Nearest Neighbor Gaussian Process

TL;DR

This work proposes variational nearest neighbor Gaussian process (VNNGP), which introduces a prior that only retains correlations within nearest-neighboring observations, thereby inducing sparse precision structure and comparing VNNGP to other scalable GPs.

Abstract

Variational approximations to Gaussian processes (GPs) typically use a small set of inducing points to form a low-rank approximation to the covariance matrix. In this work, we instead exploit a sparse approximation of the precision matrix. We propose variational nearest neighbor Gaussian process (VNNGP), which introduces a prior that only retains correlations within nearest-neighboring observations, thereby inducing sparse precision structure. Using the variational framework, VNNGP's objective can be factorized over both observations and inducing points, enabling stochastic optimization with a time complexity of . Hence, we can arbitrarily scale the inducing point size, even to the point of putting inducing points at every observed location. We compare VNNGP to other scalable GPs through various experiments, and demonstrate that VNNGP (1) can dramatically outperform low-rank methods, and (2) is less prone to overfitting than other nearest neighbor methods.
Paper Structure (37 sections, 1 theorem, 25 equations, 8 figures, 3 tables)

This paper contains 37 sections, 1 theorem, 25 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Gaussian Processes
  4. Sparse Precision Approximations
  5. Stochastic Variational Gaussian Processes
  6. Other Related Works
  7. Variational GP inference
  8. Nearest neighbor approximations.
  9. Variational Nearest Neighbor GP
  10. Ordering of inducing points.
  11. Variational family.
  12. Optimization objective
  13. Computing the Data Likelihood Term
  14. Computing the KL Divergence
  15. Advantages
  16. ...and 22 more sections

Key Result

Proposition 3.1

Let $p(\mathbf{u})$ and $q(\mathbf{u})$ be two distributions for an $M$-dimensional random variable $\mathbf{u}$, and let $\mathbf{u}_1$ be any sub-vector of $\mathbf{u}$, then

Figures (8)

  • Figure 1: The Cholesky factor of prior precision matrix for $M=20$ inducing points (with even spacing of 1) by VNNGP with different number of nearest neighbors $K$. We use a squared exponential kernel with outputscale 1 and lengthscale 1. For each row of the Cholesky factor, at most $K+1$ elements are non-zero, and when $K=10$ the approximation is qualitatively close to the exact one.
  • Figure 2: Test set RMSE and NLL as a function of number of nearest neighbors $K$ for VNNGP (green) on Elevators (left column) and UKHousing datasets (right column). Results for SVGP (blue dashed) and exact GP (purple dotted) are included as baselines.
  • Figure 3: KL divergence computed by VNNGP (green) and SWSGP (orange) as a function of $K$ for different prior kernel lengthscales. Both methods tend to underestimate the KL (SWSGP provably), and converges to the exact value computed by SVGP (blue dotted) as $K \rightarrow M$. Morever, VNNGP's KL is closer to the exact one compared to SWSGP for the same $K$ across all cases.
  • Figure 4: Posterior by Exact GP (purple), SVGP (blue), VNNGP (green) and SWSGP (orange), with 95% uncertainty interval. The grey line is to the true data generating process, and scattering black dots are highly noisy observations.
  • Figure 5: Negative training ELBO (solid line) and test NLL (dotted line) as a function of the likelihood noise by SVGP (blue), VNNGP (green) and SWSGP (orange) on UKHousing dataset. All ELBO and NLL values are scaled to $[0,1]$. The x-axis is in log-scale. The noise value picked by maximizing ELBO for each method is indicated by the vertical dashed line with corresponding color. The vertical purple line indicates the value ($0.28$) learned by exact GP.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Proposition 3.1
  • proof