Wasserstein Barycenter Gaussian Process based Bayesian Optimization

TL;DR

This work addresses instability and misspecification in GP-based Bayesian optimization arising from online GP hyperparameter fitting. It proposes WBGP-BO, which fixes a set of kernel hyperparameters, fits multiple GPs, and aggregates them into a single surrogate via the Wasserstein barycenter, yielding a posterior with mean and uncertainty that reflect an ensemble. The authors prove that the WBGP-LCB equals the average of individual GPs' LCBs under equal acquisition weights and show WBGP-BO behaves like a principled GP ensemble. Empirical results on 1D benchmark problems demonstrate that WBGP-BO often matches GP-BO on easy problems and significantly improves performance on harder ones, with limited sensitivity to the number of included GPs; future work includes extending to higher dimensions and comparing with additional baselines.

Abstract

Gaussian Process based Bayesian Optimization is a widely applied algorithm to learn and optimize under uncertainty, well-known for its sample efficiency. However, recently -- and more frequently -- research studies have empirically demonstrated that the Gaussian Process fitting procedure at its core could be its most relevant weakness. Fitting a Gaussian Process means tuning its kernel's hyperparameters to a set of observations, but the common Maximum Likelihood Estimation technique, usually appropriate for learning tasks, has shown different criticalities in Bayesian Optimization, making theoretical analysis of this algorithm an open challenge. Exploiting the analogy between Gaussian Processes and Gaussian Distributions, we present a new approach which uses a prefixed set of hyperparameters values to fit as many Gaussian Processes and then combines them into a unique model as a Wasserstein Barycenter of Gaussian Processes. We considered both "easy" test problems and others known to undermine the \textit{vanilla} Bayesian Optimization algorithm. The new method, namely Wasserstein Barycenter Gausssian Process based Bayesian Optimization (WBGP-BO), resulted promising and able to converge to the optimum, contrary to vanilla Bayesian Optimization, also on the most "tricky" test problems.

Figures (5)

• Figure 1: Shape preservation of the Wasserstein barycenter against L2-norm barycenter.
• Figure 2: Two different GPs fitted to the same set of five observations. On the vertical line: the two Gaussian distributions (blue and green) for posterior about $f(\boldsymbol{x})$, at that specific location, and their associated Wasserstein barycenter (purple, dashed).
• Figure 3: The Wasserstein Barycenter Gaussian Process (WBGP) obtained by computing the Wasserstein barycenter of the two GPs in Figure \ref{['fig:WBGP_x']}, $\forall\;\boldsymbol{x} \in \mathcal{X}$.
• Figure 4: Graphical representation of the test problems reported in Table \ref{['tab:1']} (the original search space of each function has been rescaled in $[0,1]$).
• Figure 5: Best observed values over the sequential queries (average and standard deviation over the 30 independent runs). WBGPBO with $N=16$ is omitted because almost always overlapping WBGP-BO with $N=16$ or being slightly worse.