Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB

TL;DR

This study tackles the challenge of scaling Bayesian Optimization to high-dimensional, low-budget black-box problems by benchmarking five high-dimensional BO variants—SAASBO, RDUCB, PCA-BO, KPCA-BO, and TuRBO—against vanilla BO and CMA-ES on the 24 BBOB functions across dimensions up to $D=60$ with a budget of $10D+50$. It demonstrates that trust-region based TuRBO often provides the best overall performance as dimensionality increases, while vanilla BO excels at very small budgets and CMA-ES can catch up with more evaluations; PCA-BO and KPCA-BO offer fast initial convergence on certain landscapes. The results reveal substantial landscape- and budget-dependent variation, highlight the CPU-time trade-offs (notably the high cost of SAASBO), and suggest that hybridizing linear embeddings with trust regions could yield robust, scalable optimizers for expensive evaluations. The work provides reproducible benchmarking data and points toward practical guidance for algorithm selection and future hybrid designs in high-dimensional Bayesian optimization.

Abstract

Bayesian Optimization (BO) is a class of black-box, surrogate-based heuristics that can efficiently optimize problems that are expensive to evaluate, and hence admit only small evaluation budgets. BO is particularly popular for solving numerical optimization problems in industry, where the evaluation of objective functions often relies on time-consuming simulations or physical experiments. However, many industrial problems depend on a large number of parameters. This poses a challenge for BO algorithms, whose performance is often reported to suffer when the dimension grows beyond 15 variables. Although many new algorithms have been proposed to address this problem, it is not well understood which one is the best for which optimization scenario. In this work, we compare five state-of-the-art high-dimensional BO algorithms, with vanilla BO and CMA-ES on the 24 BBOB functions of the COCO environment at increasing dimensionality, ranging from 10 to 60 variables. Our results confirm the superiority of BO over CMA-ES for limited evaluation budgets and suggest that the most promising approach to improve BO is the use of trust regions. However, we also observe significant performance differences for different function landscapes and budget exploitation phases, indicating improvement potential, e.g., through hybridization of algorithmic components.

Figures (22)

• Figure 1: Illustration of the five different categories of BO-based algorithms for high-dimensional problems. The advantages and disadvantages of each category are listed, along with the algorithm chosen for our comparison.
• Figure 2: The best-so-far target gap for dimension 10.
• Figure 3: CPU time in seconds (logarithmic scale) for the entire run (left) and model fitting and acquisition function optimization (right) in dimension 10. Values for the total CPU time are averaged across all 24 BBOB functions. Model fitting time and acquisition function optimization time are first averaged over all iterations of one run, and then across the 24 BBOB functions. The black line in each bar represents the bootstrap confidence interval.
• Figure 4: The best-so-far target gap for dimension 20.
• Figure 5: CPU time in seconds (logarithmic scale) for the entire run (left) and model fitting and acquisition function optimization (right) in dimension 20. Values for the total CPU time are averaged across all 24 BBOB functions. Model fitting time and acquisition function optimization time are first averaged over all iterations of one run, and then across the 24 BBOB functions. The black line in each bar represents the bootstrap confidence interval.