Gaussian process interpolation with conformal prediction: methods and comparative analysis

TL;DR

The paper addresses calibration of prediction intervals in Gaussian process interpolation when hyperparameters are chosen by ML. It adopts conformal prediction (CP) methods—Full CP (FCP), Split CP (SCP), Jackknife CP (JCP), and Jackknife+ (J+)—and adapts them to GP interpolation, introducing two GP-specific variants: Full-Conformal Prediction for GP (FCP-GP) and Jackknife+ for GP (J+GP). A new asymmetric-score CP variant (asymJ+GP) is proposed to better handle skewed predictive distributions. Numerical experiments on several test functions show CP methods substantially improve calibration (lower IAE) with competitive RMSE, indicating that CP can enhance uncertainty quantification in GP interpolation without sacrificing accuracy. The work advocates for broader adoption of CP in the GP community and highlights practical variants tuned for noise-free GP interpolation and potential skewness in predictions.

Abstract

This article advocates the use of conformal prediction (CP) methods for Gaussian process (GP) interpolation to enhance the calibration of prediction intervals. We begin by illustrating that using a GP model with parameters selected by maximum likelihood often results in predictions that are not optimally calibrated. CP methods can adjust the prediction intervals, leading to better uncertainty quantification while maintaining the accuracy of the underlying GP model. We compare different CP variants and introduce a novel variant based on an asymmetric score. Our numerical experiments demonstrate the effectiveness of CP methods in improving calibration without compromising accuracy. This work aims to facilitate the adoption of CP methods in the GP community.

Figures (2)

• Figure 1: IAE versus RMSE, computed by leave-one-out (LOO) on the left-hand side, and on a test set of $1500$ points on the right-hand side. Each red point correspond to a different value for the variance and the range/lenghscale parameters of the covariance of $Z$. The blue square represents the parameter selected by REML, and the green star represents the metrics computed when a conformal prediction method (Jackknife+ for GP) is used to build the prediction intervals. Values, in LOO, above $J_{\rm IAE} > 0.3$ and RMSE $> 5\cdot10^4$ are not shown.
• Figure 2: Coverage and average width of the prediction intervals at level $0.9$ for the Goldstein-Price function with $40$ training points. The GP model parameters $\sigma$ and $\rho$ are selected by REML. The intervals are computed using the posterior variance, the FCP-GP method, the J+GP method, and the asymJ+GP method.