Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit Manifold Gaussian Process Regression

Bernardo Fichera, Viacheslav Borovitskiy, Andreas Krause, Aude Billard

TL;DR

The paper tackles high‑dimensional Gaussian process regression by learning an implicit low‑dimensional manifold from labeled and unlabeled data. It introduces IMGP, a differentiable framework that builds a geometry‑aware prior via graph Matérn kernels on a learned manifold and blends it with a standard Euclidean GP through a distance‑dependent bump. The method leverages KNN graphs, Nyström extensions, Lanczos‑based eigenpairs, and efficient matrix‑vector operations to scale to large datasets, with theoretical grounding in convergence to manifold Matérn kernels. Empirically, IMGP improves predictive uncertainty and calibration on synthetic and real high‑dimensional tasks, particularly in semi‑supervised settings, while highlighting sensitivity to graph quality and approximation choices. This work advances probabilistic modeling in high dimensions by enabling geometry learning from data and integrating it into scalable, differentiable GP inference.

Abstract

Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this technique to higher dimensions is to leverage the implicit low-dimensional manifold upon which the data actually lies, as postulated by the manifold hypothesis. Prior work ordinarily requires the manifold structure to be explicitly provided though, i.e. given by a mesh or be known to be one of the well-known manifolds like the sphere. In contrast, in this paper we propose a Gaussian process regression technique capable of inferring implicit structure directly from data (labeled and unlabeled) in a fully differentiable way. For the resulting model, we discuss its convergence to the Matérn Gaussian process on the assumed manifold. Our technique scales up to hundreds of thousands of data points, and may improve the predictive performance and calibration of the standard Gaussian process regression in high-dimensional settings.

Implicit Manifold Gaussian Process Regression

TL;DR

The paper tackles high‑dimensional Gaussian process regression by learning an implicit low‑dimensional manifold from labeled and unlabeled data. It introduces IMGP, a differentiable framework that builds a geometry‑aware prior via graph Matérn kernels on a learned manifold and blends it with a standard Euclidean GP through a distance‑dependent bump. The method leverages KNN graphs, Nyström extensions, Lanczos‑based eigenpairs, and efficient matrix‑vector operations to scale to large datasets, with theoretical grounding in convergence to manifold Matérn kernels. Empirically, IMGP improves predictive uncertainty and calibration on synthetic and real high‑dimensional tasks, particularly in semi‑supervised settings, while highlighting sensitivity to graph quality and approximation choices. This work advances probabilistic modeling in high dimensions by enabling geometry learning from data and integrating it into scalable, differentiable GP inference.

Abstract

Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this technique to higher dimensions is to leverage the implicit low-dimensional manifold upon which the data actually lies, as postulated by the manifold hypothesis. Prior work ordinarily requires the manifold structure to be explicitly provided though, i.e. given by a mesh or be known to be one of the well-known manifolds like the sphere. In contrast, in this paper we propose a Gaussian process regression technique capable of inferring implicit structure directly from data (labeled and unlabeled) in a fully differentiable way. For the resulting model, we discuss its convergence to the Matérn Gaussian process on the assumed manifold. Our technique scales up to hundreds of thousands of data points, and may improve the predictive performance and calibration of the standard Gaussian process regression in high-dimensional settings.
Paper Structure (33 sections, 4 theorems, 38 equations, 7 figures, 5 tables)

This paper contains 33 sections, 4 theorems, 38 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work and Contribution
  3. Gaussian Processes
  4. Matérn Gaussian Processes on Explicit Manifolds and Graphs
  5. Implicit Manifolds and Gaussian Processes on Them
  6. Background on Approximating the Eigenpairs of the Laplace--Beltrami Operator
  7. Approximating Matérn Kernels on Manifolds
  8. Implicit Manifold Gaussian Process
  9. Efficient Training of the Implicit Manifold Gaussian Processes
  10. Noiseless Supervised Learning
  11. Noiseless Semi-Supervised Learning
  12. Handling Noisy Observations
  13. Resulting Algorithm
  14. Experiments
  15. Synthetic Examples
  16. ...and 18 more sections

Key Result

Proposition 1

Denote the eigenpairs by $\lambda_l, f_l$ for a graph Laplacian and by $\lambda^{\c{M}}_l, f^{\c{M}}_l$ for the Laplace--Beltrami operator. Fix $\delta > 0$. Assume that, with probability at least $1 - \delta$, for all $\varepsilon > 0$, for $\alpha$ small enough and for $N$ large enough we have $|\

Figures (7)

  • Figure 1: Euclidean (standard Matérn-$5/2$ kernel) vs ours (implicit manifold) Gaussian process regression for data that lies on a dumbbell-shaped curve ($1$-dimensional manifold) assumed unknown. The data contains a small set of labeled points and a large set of unlabeled points. Our technique recognizes that the two lines in the middle are intrinsically far away from each other, giving a much better model on and near the manifold. Far away from the manifold it reverts to the Euclidean model.
  • Figure 2: Kernel values $k({\color{greypointcolor}\bullet}, \cdot)$ and samples for the Matérn-$3/2$ Gaussian processes on the sphere manifold $\mathbb{S}_2$ and for the approximating Matérn-$5/2$ process on a geodesic polyhedron graph $\f{P}\mathbb{S}_2$.
  • Figure 3: Different quantities connected to kernel extension. Notice that the values on subfigures (a) and (b) are artificially restricted to the set $\f{dist}(\cdot, \c{M}) < 3 \alpha$ to maintain numerical stability.
  • Figure 4: The ground truth function on the dumbbell manifold and the predictions of the implicit manifold Gaussian process regression (IMGP) under different levels of noise.
  • Figure 5: Root Mean Square Error (RMSE) and Negative Log-Likelihood (NLL) for increasing number of eigenpairs $L$ (left panels) and increasing fraction $f = n\%N$ of labeled points (right panels). The legend in (a) and (b) refers to the number of hyperparameter optimization iterations.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 2
  • proof
  • Proposition 2
  • proof