Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Adaptive Dropout Approach for High-Dimensional Bayesian Optimization

Jundi Huang, Dawei Zhan

TL;DR

The paper tackles the inefficiency of Bayesian optimization in high-dimensional settings by introducing AdaDropout, an adaptive dimensionality reduction method that progressively dropout variables in the acquisition function. AdaDropout starts with full dimensional optimization and reduces the active subspace when progress stalls, using a simple rule $d \leftarrow d - 1$ if $f(\\bm{x}_{\\text{next}}) > f_{\\min}$ and $d > 1$, thereby shifting from global exploration to local exploitation in lower dimensions. Empirical results on 100D CEC2013 and CEC2017 benchmarks demonstrate that AdaDropout consistently outperforms standard BO and multiple high-dimensional BO baselines in terms of convergence speed and final objective values. The work provides a practical, parameter-light approach to high-dimensional expensive optimization and points to future extensions for constrained and multi-objective problems.

Abstract

Bayesian optimization (BO) is a widely used algorithm for solving expensive black-box optimization problems. However, its performance decreases significantly on high-dimensional problems due to the inherent high-dimensionality of the acquisition function. In the proposed algorithm, we adaptively dropout the variables of the acquisition function along the iterations. By gradually reducing the dimension of the acquisition function, the proposed approach has less and less difficulty to optimize the acquisition function. Numerical experiments demonstrate that AdaDropout effectively tackle high-dimensional challenges and improve solution quality where standard Bayesian optimization methods often struggle. Moreover, it achieves superior results when compared with state-of-the-art high-dimensional Bayesian optimization approaches. This work provides a simple yet efficient solution for high-dimensional expensive optimization.

An Adaptive Dropout Approach for High-Dimensional Bayesian Optimization

TL;DR

The paper tackles the inefficiency of Bayesian optimization in high-dimensional settings by introducing AdaDropout, an adaptive dimensionality reduction method that progressively dropout variables in the acquisition function. AdaDropout starts with full dimensional optimization and reduces the active subspace when progress stalls, using a simple rule if and , thereby shifting from global exploration to local exploitation in lower dimensions. Empirical results on 100D CEC2013 and CEC2017 benchmarks demonstrate that AdaDropout consistently outperforms standard BO and multiple high-dimensional BO baselines in terms of convergence speed and final objective values. The work provides a practical, parameter-light approach to high-dimensional expensive optimization and points to future extensions for constrained and multi-objective problems.

Abstract

Bayesian optimization (BO) is a widely used algorithm for solving expensive black-box optimization problems. However, its performance decreases significantly on high-dimensional problems due to the inherent high-dimensionality of the acquisition function. In the proposed algorithm, we adaptively dropout the variables of the acquisition function along the iterations. By gradually reducing the dimension of the acquisition function, the proposed approach has less and less difficulty to optimize the acquisition function. Numerical experiments demonstrate that AdaDropout effectively tackle high-dimensional challenges and improve solution quality where standard Bayesian optimization methods often struggle. Moreover, it achieves superior results when compared with state-of-the-art high-dimensional Bayesian optimization approaches. This work provides a simple yet efficient solution for high-dimensional expensive optimization.

Paper Structure

This paper contains 14 sections, 9 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Backgrounds
  3. Gaussian Process Model
  4. Acquisition Function
  5. Bayesian Optimization
  6. Proposed Approach
  7. The Basic Idea
  8. The Computational Framework
  9. Illustrative Example
  10. Numerical Experiments
  11. Experiment Settings
  12. Results on CEC 2013
  13. Results on CEC 2017
  14. Conclusions

Figures (6)

  • Figure 1: GP approximation of the $f = \cos(x)+\sin(2x)+0.5x$ function
  • Figure 2: The EI function on the $f = \cos(x)+\sin(2x)+0.5x$ problem
  • Figure 3: Main Steps of the Standard BO and the Proposed AdaDropout
  • Figure 4: Illustration of the AdaDropout strategy. Left: The current dimensionality of the acquisition function. Middle: A grid showing the selected coordinates (blue squares) for the current iteration. Right: Comparison between $f_{\min}$ and $f(x_{\text{next}})$ used to decide whether to drop an optimization variable.
  • Figure 5: Convergence histories of the BOs and the proposed AdaDropout on the CEC2013 test problems. (a) $f_\text{1}$. (b) $f_\text{2}$. (c) $f_\text{6}$. (d) $f_\text{11}$. (e) $f_\text{17}$. (f) $f_\text{19}$.
  • ...and 1 more figures