Gaussian Process Thompson Sampling via Rootfinding

TL;DR

An efficient global optimization strategy for GP-TS that carefully selects starting points for gradient-based multi-start optimizers and optimizes the posterior sample using a differentiable, decoupled representation is introduced.

Abstract

Thompson sampling (TS) is a simple, effective stochastic policy in Bayesian decision making. It samples the posterior belief about the reward profile and optimizes the sample to obtain a candidate decision. In continuous optimization, the posterior of the objective function is often a Gaussian process (GP), whose sample paths have numerous local optima, making their global optimization challenging. In this work, we introduce an efficient global optimization strategy for GP-TS that carefully selects starting points for gradient-based multi-start optimizers. It identifies all local optima of the prior sample via univariate global rootfinding, and optimizes the posterior sample using a differentiable, decoupled representation. We demonstrate remarkable improvement in the global optimization of GP posterior samples, especially in high dimensions. This leads to dramatic improvements in the overall performance of Bayesian optimization using GP-TS acquisition functions, surprisingly outperforming alternatives like GP-UCB and EI.

Figures (2)

• Figure 1: Illustration of exploration and exploitation sets for global optimization of GP-TS acquisition functions. (a) When the global minimum of the GP-TS acquisition function lies outside the interpolation region, it is typically identified by starting the gradient-based optimizer at a local minimum of the prior sample. (b) When the global minimum is within the interpolation region, it can be found by starting the gradient-based optimizer at either a data point or a local minimum of the prior sample.
• Figure 2: Optimization results for (a) 2d Schwefel and (b) 10d Levy functions. Top-left: Cumulative distances between new candidate solutions $\mathbf{x}_k^\star$ to the true global minimums $\mathbf{x}_k^\text{t}$ of the GP-TS acquisition functions $\widetilde{f}(\mathbf{x})$ for 2d Schwefel function. Bottom-left: Cumulative optimized values $\widetilde{f}_k^\star$ for 10d Levy function. Middle: Cumulative run time $t_k$ required for optimizing $\widetilde{f}(\mathbf{x})$. Right: Histories of medians and interquartile ranges of solutions from 20 runs of our method, TS-RF, EI, and LCB.