Evolve Cost-aware Acquisition Functions Using Large Language Models

TL;DR

This work tackles expensive optimization with heterogeneous evaluation costs by introducing EvolCAF, an automated framework that designs cost-aware acquisition functions (AFs) via large language models (LLMs) within an evolutionary computation loop. By evolving explicit AFs in the algorithmic space, EvolCAF yields interpretable, high-performing strategies that leverage historical data, GP surrogates, and budget information, outperforming hand-crafted AFs like EIpu and EI-cool on 12 synthetic problems and 3 real-world hyperparameter-tuning tasks. The approach combines prompt-driven initialization, crossover, and mutation to generate elite AFs whose best instance comprises three components: a modified EI term with historical-uncertainty awareness, a budget-aware term, and a distance-based exploration modifier, all expressed explicitly. The results demonstrate strong generalization and efficiency, suggesting broad applicability to cost-aware BO settings and potential extensions to high-dimensional, batch, and multi-objective scenarios, with practical impact in domains requiring budget-conscious optimization.

Abstract

Many real-world optimization scenarios involve expensive evaluation with unknown and heterogeneous costs. Cost-aware Bayesian optimization stands out as a prominent solution in addressing these challenges. To approach the global optimum within a limited budget in a cost-efficient manner, the design of cost-aware acquisition functions (AFs) becomes a crucial step. However, traditional manual design paradigm typically requires extensive domain knowledge and involves a labor-intensive trial-and-error process. This paper introduces EvolCAF, a novel framework that integrates large language models (LLMs) with evolutionary computation (EC) to automatically design cost-aware AFs. Leveraging the crossover and mutation in the algorithmic space, EvolCAF offers a novel design paradigm, significantly reduces the reliance on domain expertise and model training. The designed cost-aware AF maximizes the utilization of available information from historical data, surrogate models and budget details. It introduces novel ideas not previously explored in the existing literature on acquisition function design, allowing for clear interpretations to provide insights into its behavior and decision-making process. In comparison to the well-known EIpu and EI-cool methods designed by human experts, our approach showcases remarkable efficiency and generalization across various tasks, including 12 synthetic problems and 3 real-world hyperparameter tuning test sets.

Figures (5)

• Figure 1: Flowchart of EvolCAF framework. The left box presents the evolution of cost-aware AFs enabled by EvolCAF, wherein each individual in the population is an AF, represented with an algorithm description and a code block implementation. The initialization, crossover, and mutation are facilitated by LLMs. The middle box shows the cost-aware BO loop, each AF is evaluated on a set of synthetic instances to calculate the optimal gap as its fitness value.
• Figure 2: Prompts used in EvolCAF for initialization, crossover, and mutation.
• Figure 3: The evolutionary process of acquisition functions.
• Figure 4: Optimal acquisition function designed by EvolCAF. The results include a linguistic description of the algorithmic idea, as well as a code implementation with annotations, all the contents are produced by LLMs.
• Figure 5: Histograms of evaluation cost frequency collected in 10 independent runs on CIFAR-10 data set with C=4000.