Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Acquisition Selection for Bayesian Optimization with Large Language Models

Giang Ngo, Dat Phan Trong, Dang Nguyen, Sunil Gupta, Svetha Venkatesh

TL;DR

LMABO reframes acquisition function selection in Bayesian Optimization as an online, in-context decision problem solvable by a pre-trained Large Language Model. By serializing the full optimization state into a structured prompt and employing a zero-shot strategist, LMABO dynamically selects AFs from a diverse portfolio, achieving robust, state-aware adaptation. Across 50 benchmarks, LMABO outperforms static, adaptive, and other LLM-based baselines, with ablations confirming that each state component and the prompting design contribute to performance. The work demonstrates that LLMs can reason over both tactical and landscape information to emulate expert BO practices, offering a practical, scalable route to improved sample efficiency in black-box optimization.

Abstract

Bayesian Optimization critically depends on the choice of acquisition function, but no single strategy is universally optimal; the best choice is non-stationary and problem-dependent. Existing adaptive portfolio methods often base their decisions on past function values while ignoring richer information like remaining budget or surrogate model characteristics. To address this, we introduce LMABO, a novel framework that casts a pre-trained Large Language Model (LLM) as a zero-shot, online strategist for the BO process. At each iteration, LMABO uses a structured state representation to prompt the LLM to select the most suitable acquisition function from a diverse portfolio. In an evaluation across 50 benchmark problems, LMABO demonstrates a significant performance improvement over strong static, adaptive portfolio, and other LLM-based baselines. We show that the LLM's behavior is a comprehensive strategy that adapts to real-time progress, proving its advantage stems from its ability to process and synthesize the complete optimization state into an effective, adaptive policy.

Adaptive Acquisition Selection for Bayesian Optimization with Large Language Models

TL;DR

LMABO reframes acquisition function selection in Bayesian Optimization as an online, in-context decision problem solvable by a pre-trained Large Language Model. By serializing the full optimization state into a structured prompt and employing a zero-shot strategist, LMABO dynamically selects AFs from a diverse portfolio, achieving robust, state-aware adaptation. Across 50 benchmarks, LMABO outperforms static, adaptive, and other LLM-based baselines, with ablations confirming that each state component and the prompting design contribute to performance. The work demonstrates that LLMs can reason over both tactical and landscape information to emulate expert BO practices, offering a practical, scalable route to improved sample efficiency in black-box optimization.

Abstract

Bayesian Optimization critically depends on the choice of acquisition function, but no single strategy is universally optimal; the best choice is non-stationary and problem-dependent. Existing adaptive portfolio methods often base their decisions on past function values while ignoring richer information like remaining budget or surrogate model characteristics. To address this, we introduce LMABO, a novel framework that casts a pre-trained Large Language Model (LLM) as a zero-shot, online strategist for the BO process. At each iteration, LMABO uses a structured state representation to prompt the LLM to select the most suitable acquisition function from a diverse portfolio. In an evaluation across 50 benchmark problems, LMABO demonstrates a significant performance improvement over strong static, adaptive portfolio, and other LLM-based baselines. We show that the LLM's behavior is a comprehensive strategy that adapts to real-time progress, proving its advantage stems from its ability to process and synthesize the complete optimization state into an effective, adaptive policy.
Paper Structure (37 sections, 1 equation, 3 figures, 12 tables, 1 algorithm)

This paper contains 37 sections, 1 equation, 3 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works
  4. Portfolio-Based Strategies
  5. Learning-Based Strategies
  6. LLM-Based Bayesian Optimization
  7. Methodology
  8. Language Model-assisted Adaptive Bayesian Optimization (LMABO)
  9. The LLM as a Zero-Shot Strategist
  10. Optimization State Representation
  11. Experiments
  12. Experiment Setup
  13. Baselines
  14. Benchmark Problems
  15. Implementation Details
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Impact of task-specific context on LMABO performance. Results are averaged over 10 runs with standard deviation shown as shaded regions.
  • Figure 2: LMABO's acquisition function selection behaviors. Note that these behaviors are aggregated across all runs on all problems.
  • Figure 3: Word frequency in justifications. Red box (bottom left) is the explorative group. Blue box (top right) is the exploitative group.