FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

TL;DR

FunBO tackles the problem of finding acquisition functions that are robust across diverse optimization problems by learning AFs as code with an LLM, guided by a FunSearch-inspired discovery loop. It demonstrates that AFs discovered in this way generalize within and across function classes and can outperform general-purpose AFs while nearing performance of function-specific, transfer-learned AFs. The method emphasizes interpretability and practical deployability by returning executable AF code and using a population-based, islanded program database. Overall, FunBO broadens the applicability of AF design in Bayesian optimization and highlights the potential of LLM-driven algorithm discovery for sampling policy design.

Abstract

The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

Figures (21)

• Figure 1: Left: The funbo algorithm. Right: Graphical representation of funbo. The different funbo component w.r.t. funsearch romera2023mathematical are highlighted in color.)
• Figure 2: Top: funbo's initial af takes the functional form of ei with inputs given by the posterior parameter of the gp at a set of potential sample locations, the incumbent and a parameter $\beta=1$. Bottom: funbo prompt includes two previously generated afs which are sampled from db and are sorted in ascending order based on the score achieved on $\mathcal{G}_{\textsc{T}\text{r}}$. The llm generates a third af, acquisition_function_v2, representing an improved version of the highest scoring program.
• Figure 3: ood-bench. Left: Code for $\alpha_{\textsc{f}\text{un}\textsc{bo}\xspace}$. Right: Different afs trading-off exploration and exploitation for two one-dimensional objective functions (green lines). Blue and gray trajectories track the points queried by $\alpha_{\textsc{f}\text{un}\textsc{bo}\xspace}$, ei and ucb over 150 steps (right $y$-axis). All afs behave similarly for Styblinski-Tang (top, note that trajectories are overlapping), converging to the true optimizer (red vertical line) in fewer than 25 trials. Instead, for Weierstrass (bottom), ei and ucb get stuck after a few trials while $\alpha_{\textsc{f}\text{un}\textsc{bo}\xspace}$ continues to explore, eventually converging to the ground truth optimum.
• Figure 4: ood-bench. Average bo performance when using known general purpose afs and $\alpha_{\textsc{f}\text{un}\textsc{bo}\xspace}$. Shaded area gives $\pm$ standard deviations$/2$. The red line gives $\bar{R}_t = 0$, i.e. zero average regret.
• Figure 5: id-bench. Average bo performance when using known general purpose afs (gray lines), the af learned by metabo (black dashed line) and $\alpha_{\textsc{f}\text{un}\textsc{bo}\xspace}$ (blue line) on 100 function instances. Shaded area gives $\pm$ standard deviations$/2$. The red line represents $\bar{R}_t=0$, i.e. zero average regret.