BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique

TL;DR

BOOST addresses the challenge of selecting both kernel and acquisition-function hyperparameters in Bayesian Optimization by performing offline, data-in-hand based retrospective evaluations to identify the kernel–acquisition pair with the fastest convergence. It partitions the available data into a reference set and a query set using K-means, then simulates internal BO runs for each candidate pair to measure how quickly the target is reached, selecting the best pair for the actual optimization. This data-driven, iteration-by-iteration selection avoids relying on uncertain surrogate predictions and eliminates manual tuning, showing robust gains over fixed-hyperparameter BO and competitive state-of-the-art adaptive methods on synthetic and real-world HPO tasks. The approach offers practical benefits for expensive black-box problems by systematically leveraging existing evaluations to guide hyperparameter choice without extra evaluations or training requirements.

Abstract

The performance of Bayesian optimization (BO), a highly sample-efficient method for expensive black-box problems, is critically governed by the selection of its hyperparameters, including the kernel and acquisition functions. This presents a significant practical challenge: an inappropriate combination can lead to poor performance and wasted evaluations. While individual improvements to kernel functions (e.g., tree-based kernels, deep kernel learning) and acquisition functions (e.g., multi-step lookahead, tree-based planning) have been actively explored, the joint and autonomous selection of the best pair has been largely overlooked, forcing practitioners to rely on heuristics or costly manual tuning. We propose BOOST (Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique), a novel framework that automates this selection. BOOST utilizes a lightweight, offline evaluation stage to predict the performance of various kernel-acquisition pairs and identify the most promising pair before committing to expensive evaluations. Using K-means clustering, BOOST first selects initial subsets from previously observed data-in-hand and prepares all possible kernel-acquisition pairs from user-chosen candidates. For each pair, BOOST conducts internal BO runs starting with the initial subset, evaluating how many iterations are required to find the target value within the remaining data, thereby identifying the pair with the best retrospective performance for future optimization. Experiments on synthetic benchmarks and real-world hyperparameter optimization tasks demonstrate that BOOST consistently outperforms standard BO with fixed hyperparameters and state-of-the-art adaptive methods, highlighting its effectiveness and robustness in diverse problem landscapes.

Figures (7)

• Figure 1: An Overview of BOOST architecture.
• Figure 2: Optimization performance of BOOST and 16 fixed-kernel-acquisition methods on synthetic benchmark functions and HPO-B datasets. HPO-B tasks include Decision Trees (6D, space IDs: 5636, 5859), Random Forests (9D, space IDs: 7607, 7609), and XGBoost (16D, space ID: 5906). Each experiment uses 10 initial points followed by 90 optimization iterations (100 total evaluations).
• Figure 3: Optimization performance comparison between BOOST and state-of-the-art/adaptive methods on the same benchmark tasks as Figure \ref{['fig2:results']}.
• Figure 4: Comparison of BOOST performance under different data partitioning strategies
• Figure 5: Optimization performance of BOOST with different $|r_n| : |q_n|$ ratios.