Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization

TL;DR

This work develops a principled link between gradient-based local search and upper-confidence-bound minimization within local Bayesian optimization. By substituting the gradient step with a UCB-minimization step, the proposed MinUCB achieves comparable convergence to existing methods while better exploiting GP information; its look-ahead extension LA-MinUCB further improves local exploration and can be Bayes-optimal in a one-step setting. The authors provide theoretical convergence guarantees for MinUCB under common kernels and validate the methods on synthetic and reinforcement learning benchmarks, showing robust, efficient local optimization in high dimensions. Collectively, the paper advances local BO design by introducing UCB-centric descent and look-ahead strategies that outperform traditional gradient-based local methods in GP-surrogate settings.

Abstract

Local Bayesian optimization is a promising practical approach to solve the high dimensional black-box function optimization problem. Among them is the approximated gradient class of methods, which implements a strategy similar to gradient descent. These methods have achieved good experimental results and theoretical guarantees. However, given the distributional properties of the Gaussian processes applied on these methods, there may be potential to further exploit the information of the Gaussian processes to facilitate the BO search. In this work, we develop the relationship between the steps of the gradient descent method and one that minimizes the Upper Confidence Bound (UCB), and show that the latter can be a better strategy than direct gradient descent when a Gaussian process is applied as a surrogate. Through this insight, we propose a new local Bayesian optimization algorithm, MinUCB, which replaces the gradient descent step with minimizing UCB in GIBO. We further show that MinUCB maintains a similar convergence rate with GIBO. We then improve the acquisition function of MinUCB further through a look ahead strategy, and obtain a more efficient algorithm LA-MinUCB. We apply our algorithms on different synthetic and real-world functions, and the results show the effectiveness of our method. Our algorithms also illustrate improvements on local search strategies from an upper bound perspective in Bayesian optimization, and provides a new direction for future algorithm design.

Figures (3)

• Figure 1: This function $f$ is sampled from $GP(0,k(x,x^{'}))$, where $k(x,x^{'})=exp(-\frac{1}{4}(x-x^{'})^2)$, with standard derivation of white noise $\sigma=0.05$. The dataset contains 2 points, which is marked as black hollow circle. We attempt to search the next point from $\mathbf{x}_0$. The left figure shows that UCB bound is much tighter than other two gradient based bounds, and the minimum points of UCB has the best performance. This shows that minimizing UCB in this example can achieve a much better move to lower point than the gradient descent approach. The right figure illustrates UCB across the design space. Here we see that it is small only near the sampled point, and increases as it moves further away, indicating that minimizing UCB can be viewed as local strategy.
• Figure 2: Progressive optimized reward on high-dimensional synthetic functions. LA-MinUCB demonstrates fast and accurate convergence compared to other methods.
• Figure 3: Progressive optimized reward on the MuJuCo tasks. LA-MinUCB has consistently optimal performance.

Theorems & Definitions (32)

• Definition 1
• Theorem 1
• Theorem 2
• Lemma 1: Lederer et al.lederer2019uniform
• Theorem 3: Smoothness of Gaussian process
• proof
• Theorem 4
• proof
• Theorem 5: Smoothness of mean function
• proof