Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FigBO: A Generalized Acquisition Function Framework with Look-Ahead Capability for Bayesian Optimization

Hui Chen, Xuhui Fan, Zhangkai Wu, Longbing Cao

TL;DR

FigBO introduces a plug-and-play framework that equips myopic Bayesian optimization acquisitions with look-ahead by incorporating a global information gain term $\\Gamma(\\mathbf{x})$, scaled by $\\lambda=\\eta/n$. The method combines a standard base acquisition $\\alpha(\\mathbf{x})$ with the look-ahead term to guide point selection as $\\mathbf{x}_{n+1} \\in \\arg\\max (\\alpha(\\mathbf{x})+\\lambda\\Gamma(\\mathbf{x}))$, and computes $\\Gamma(\\mathbf{x})$ via Monte Carlo with Sherman-Morrison updates for efficiency. Theoretical results show that the regret of the FigBO-augmented EI, $EI_{\\Gamma,n}$, is asymptotically equivalent to vanilla EI, preserving convergence rates, while empirically FigBO delivers state-of-the-art performance across GP-prior samples, synthetic benchmarks, and MLP hyperparameter optimization, often achieving faster convergence than non-myopic counterparts. The approach offers a practical balance between exploration and exploitation, bridging the gap between myopic efficiency and non-myopic performance, with broad applicability to typical surrogate models and acquisition strategies.

Abstract

Bayesian optimization is a powerful technique for optimizing expensive-to-evaluate black-box functions, consisting of two main components: a surrogate model and an acquisition function. In recent years, myopic acquisition functions have been widely adopted for their simplicity and effectiveness. However, their lack of look-ahead capability limits their performance. To address this limitation, we propose FigBO, a generalized acquisition function that incorporates the future impact of candidate points on global information gain. FigBO is a plug-and-play method that can integrate seamlessly with most existing myopic acquisition functions. Theoretically, we analyze the regret bound and convergence rate of FigBO when combined with the myopic base acquisition function expected improvement (EI), comparing them to those of standard EI. Empirically, extensive experimental results across diverse tasks demonstrate that FigBO achieves state-of-the-art performance and significantly faster convergence compared to existing methods.

FigBO: A Generalized Acquisition Function Framework with Look-Ahead Capability for Bayesian Optimization

TL;DR

FigBO introduces a plug-and-play framework that equips myopic Bayesian optimization acquisitions with look-ahead by incorporating a global information gain term , scaled by . The method combines a standard base acquisition with the look-ahead term to guide point selection as , and computes via Monte Carlo with Sherman-Morrison updates for efficiency. Theoretical results show that the regret of the FigBO-augmented EI, , is asymptotically equivalent to vanilla EI, preserving convergence rates, while empirically FigBO delivers state-of-the-art performance across GP-prior samples, synthetic benchmarks, and MLP hyperparameter optimization, often achieving faster convergence than non-myopic counterparts. The approach offers a practical balance between exploration and exploitation, bridging the gap between myopic efficiency and non-myopic performance, with broad applicability to typical surrogate models and acquisition strategies.

Abstract

Bayesian optimization is a powerful technique for optimizing expensive-to-evaluate black-box functions, consisting of two main components: a surrogate model and an acquisition function. In recent years, myopic acquisition functions have been widely adopted for their simplicity and effectiveness. However, their lack of look-ahead capability limits their performance. To address this limitation, we propose FigBO, a generalized acquisition function that incorporates the future impact of candidate points on global information gain. FigBO is a plug-and-play method that can integrate seamlessly with most existing myopic acquisition functions. Theoretically, we analyze the regret bound and convergence rate of FigBO when combined with the myopic base acquisition function expected improvement (EI), comparing them to those of standard EI. Empirically, extensive experimental results across diverse tasks demonstrate that FigBO achieves state-of-the-art performance and significantly faster convergence compared to existing methods.
Paper Structure (23 sections, 4 theorems, 23 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 23 sections, 4 theorems, 23 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Gaussian Processes
  4. Acquisition Functions
  5. Methodology
  6. Global Information Gain
  7. FigBO Algorithm
  8. Theoretical Analysis
  9. Experiment
  10. GP Prior Samples
  11. Synthetic Test Functions
  12. MLP Classification Tasks
  13. Universality of FigBO
  14. Effect of MC Samples
  15. Effect of Hyperparameter $\eta$
  16. ...and 8 more sections

Key Result

Theorem 1

For the $\mathcal{D}_n$, $\mathcal{H}_{\ell}(\mathcal{X})$, $R$ defined above, the loss $\mathcal{L}_n$ of $EI_{\Gamma,n}$ with a compact set $\mathcal{X}$ at iteration $n$ can satisfy where $C'=\frac{\tau(R/\sigma)}{\tau(-R/\sigma)} \underset{x\in \mathcal{X}}{\max} \lambda \Gamma(\mathbf{x})$.

Figures (7)

  • Figure 1: The change in the global uncertainty when a candidate point is added to the observations. When the iteration $n=4$, we add a candidate point (green point) to the observations (black points) and measure the change of the global uncertainty (red region). The candidate point that maximizes the global uncertainty change will be selected as the next query point.
  • Figure 2: Average log regret of all baselines on GP prior sample tasks across different dimensions (2D, 4D, 6D, 12D) over 20 repetitions. The average log regret and its standard errors are displayed for each acquisition function.
  • Figure 3: Average log regret of all baselines on three different synthetic functions over 20 repetitions. The average log regret and its standard errors are displayed for each acquisition function.
  • Figure 4: Best observed accuracy of all baselines on four different MLP classification tasks over 50 repetitions. The average accuracy and its standard errors are displayed for each acquisition function.
  • Figure 5: Average log regret of diverse myopic acquisition functions and their FigBO variants on three different synthetic functions over 20 repetitions. The average log regret and its standard errors are displayed for each acquisition function.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Remark 1
  • Corollary 1
  • Remark 2
  • proof
  • Lemma 1
  • Lemma 2
  • proof