Relaxed Gaussian process interpolation: a goal-oriented approach to Bayesian optimization
Sébastien Petit, Julien Bect, Emmanuel Vazquez
TL;DR
The paper introduces relaxed Gaussian process interpolation (reGP), a goal-oriented approach that relaxes interpolation constraints outside a region of interest to yield sharper predictive distributions where low function values matter, notably in Bayesian optimization. It formalizes reGP as a Gaussian predictive model conditioned on the mode of relaxed constraints, and couples hyperparameter estimation with relaxation via joint likelihood optimization, including extensions to noisy data. A truncated CRPS-based method (tCRPS) is proposed to automatically select the relaxation range, and theoretical convergence guarantees are provided for EI-based optimization when the target function lies in the RKHS of the underlying covariance. Empirical benchmarks show that reGP can substantially improve optimization performance on challenging functions, at the cost of additional computation, and the framework is extended to noisy settings and UCB-based strategies with open-source implementations available. The work highlights a practical, theory-backed pathway to goal-oriented probabilistic modeling that prioritizes predictive quality in regions of interest for improved sequential decision making.
Abstract
This work presents a new procedure for obtaining predictive distributions in the context of Gaussian process (GP) modeling, with a relaxation of the interpolation constraints outside ranges of interest: the mean of the predictive distributions no longer necessarily interpolates the observed values when they are outside ranges of interest, but are simply constrained to remain outside. This method called relaxed Gaussian process (reGP) interpolation provides better predictive distributions in ranges of interest, especially in cases where a stationarity assumption for the GP model is not appropriate. It can be viewed as a goal-oriented method and becomes particularly interesting in Bayesian optimization, for example, for the minimization of an objective function, where good predictive distributions for low function values are important. When the expected improvement criterion and reGP are used for sequentially choosing evaluation points, the convergence of the resulting optimization algorithm is theoretically guaranteed (provided that the function to be optimized lies in the reproducing kernel Hilbert space attached to the known covariance of the underlying Gaussian process). Experiments indicate that using reGP instead of stationary GP models in Bayesian optimization is beneficial.