Calibrated Computation-Aware Gaussian Processes
Disha Hegde, Mohamed Adil, Jon Cockayne
TL;DR
This work tackles the scalability of Gaussian process regression by employing computation-aware GPs (CAGPs) that incorporate uncertainty from reduced computation via probabilistic solvers. It identifies calibration issues with standard BayesCG-based CAGPs and proves that using a calibrated solver yields a calibrated CAGP posterior, introducing a Gauss-Seidel-based CAGP (CAGP-GS) within a PSIM framework. The paper develops the theoretical calibration guarantees, implements CAGP-GS, and demonstrates improved mean convergence and calibrated uncertainty on synthetic data, UCI benchmarks, and a large-scale ERA5 regression task. The findings suggest that calibrated, PSIM-driven CAGPs enable reliable, computation-efficient GP regression, especially when only a small number of iterations are feasible. This has practical impact for large-scale Bayesian regression where exact GP inference is prohibitive but calibrated uncertainty is essential.
Abstract
Gaussian processes are notorious for scaling cubically with the size of the training set, preventing application to very large regression problems. Computation-aware Gaussian processes (CAGPs) tackle this scaling issue by exploiting probabilistic linear solvers to reduce complexity, widening the posterior with additional computational uncertainty due to reduced computation. However, the most commonly used CAGP framework results in (sometimes dramatically) conservative uncertainty quantification, making the posterior unrealistic in practice. In this work, we prove that if the utilised probabilistic linear solver is calibrated, in a rigorous statistical sense, then so too is the induced CAGP. We thus propose a new CAGP framework, CAGP-GS, based on using Gauss-Seidel iterations for the underlying probabilistic linear solver. CAGP-GS performs favourably compared to existing approaches when the test set is low-dimensional and few iterations are performed. We test the calibratedness on a synthetic problem, and compare the performance to existing approaches on a large-scale global temperature regression problem.