Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings

Lam Ngo, Huong Ha, Jeffrey Chan, Hongyu Zhang

TL;DR

This work tackles high-dimensional, expensive black-box optimization by introducing BOIDS, a Bayesian Optimization framework that uses incumbent-guided direction lines and a subspace embedding strategy to enable efficient line-based search. The method combines PSO-inspired direction lines, a Thompson Sampling multi-armed bandit for line selection, and BAxUS subspace embeddings, culminating in a cohesive algorithm with local simple-regret bounds and global convergence guarantees when the embedding can contain the optimum. Theoretical analysis provides a sub-linear simple regret bound, and empirical results across synthetic and real-world tasks show BOIDS outperforming state-of-the-art baselines in both speed and accuracy. The approach offers a scalable, principled path for high-dimensional BO with practical impact in hyperparameter tuning, robotics, and engineering design.

Abstract

When it comes to expensive black-box optimization problems, Bayesian Optimization (BO) is a well-known and powerful solution. Many real-world applications involve a large number of dimensions, hence scaling BO to high dimension is of much interest. However, state-of-the-art high-dimensional BO methods still suffer from the curse of dimensionality, highlighting the need for further improvements. In this work, we introduce BOIDS, a novel high-dimensional BO algorithm that guides optimization by a sequence of one-dimensional direction lines using a novel tailored line-based optimization procedure. To improve the efficiency, we also propose an adaptive selection technique to identify most optimal lines for each round of line-based optimization. Additionally, we incorporate a subspace embedding technique for better scaling to high-dimensional spaces. We further provide theoretical analysis of our proposed method to analyze its convergence property. Our extensive experimental results show that BOIDS outperforms state-of-the-art baselines on various synthetic and real-world benchmark problems.

BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings

TL;DR

This work tackles high-dimensional, expensive black-box optimization by introducing BOIDS, a Bayesian Optimization framework that uses incumbent-guided direction lines and a subspace embedding strategy to enable efficient line-based search. The method combines PSO-inspired direction lines, a Thompson Sampling multi-armed bandit for line selection, and BAxUS subspace embeddings, culminating in a cohesive algorithm with local simple-regret bounds and global convergence guarantees when the embedding can contain the optimum. Theoretical analysis provides a sub-linear simple regret bound, and empirical results across synthetic and real-world tasks show BOIDS outperforming state-of-the-art baselines in both speed and accuracy. The approach offers a scalable, principled path for high-dimensional BO with practical impact in hyperparameter tuning, robotics, and engineering design.

Abstract

When it comes to expensive black-box optimization problems, Bayesian Optimization (BO) is a well-known and powerful solution. Many real-world applications involve a large number of dimensions, hence scaling BO to high dimension is of much interest. However, state-of-the-art high-dimensional BO methods still suffer from the curse of dimensionality, highlighting the need for further improvements. In this work, we introduce BOIDS, a novel high-dimensional BO algorithm that guides optimization by a sequence of one-dimensional direction lines using a novel tailored line-based optimization procedure. To improve the efficiency, we also propose an adaptive selection technique to identify most optimal lines for each round of line-based optimization. Additionally, we incorporate a subspace embedding technique for better scaling to high-dimensional spaces. We further provide theoretical analysis of our proposed method to analyze its convergence property. Our extensive experimental results show that BOIDS outperforms state-of-the-art baselines on various synthetic and real-world benchmark problems.

Paper Structure

This paper contains 52 sections, 3 theorems, 20 equations, 3 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Bayesian Optimization
  4. Line-based Bayesian Optimization
  5. Subspace Embedding
  6. Related Work
  7. Effective Surrogate Models.
  8. Subspace Embedding.
  9. Variable Selection.
  10. Search Space Partitioning.
  11. Objective Function Decomposition.
  12. Evolutionary Algorithms (EA).
  13. BOIDS: High-Dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings
  14. Incumbent-guided Direction Lines
  15. Adaptive Line Selection
  16. ...and 37 more sections

Key Result

Lemma 1

Let $\mathbf{r}_1, \mathbf{r}_2\in \mathbb{R}^d$ be two random vectors sampled from a uniform distribution $\mathcal{U}([0,1]^d)$, $\mathbf{h}_1, \mathbf{h}_2 \in \mathbb{R}^d$ are arbitrary fixed vectors, and the element-wise multiplications $\mathbf{v}_1 = \mathbf{r}_1 \circ \mathbf{h}_1$ and $\ma where constant $C\in \mathbb{R}$ is a constant such that $C \le \frac{1}{4} ({\Vert \mathbf{h}_1 \V

Figures (3)

  • Figure 1: Illustration of our proposed BOIDS algorithm. (1) The incumbent-guided lines are constructed from the personal and global incumbents. (2) The optimal lines are selected based on Thompson Sampling MAB strategy. (3) Line-based Optimization is performed following the optimal line. (4) Subspace embedding technique is incorporated to enhance the performance.
  • Figure 2: Comparison of our proposed method, BOIDS, against the state-of-the-art baselines on six minimization problems. Note that some methods (ALEBO, SAASBO and RDUCB) only have limited iterations due to the prohibitively high computational cost and memory required. Overall, BOIDS outperforms all baselines significantly on most problems.
  • Figure 3: Impact of BOIDS components on performance when each is alternatively removed. The incumbent-guided line component (Sec. \ref{['sec-method:incumbent-line']}) and the tailored line-based optimization (Sec. \ref{['sec-method:line-based-optimization']}) have the most significant influence.

Theorems & Definitions (7)

  • Lemma 1: Incumbent-guided Search Direction
  • Proposition 1
  • Theorem 1
  • proof : Proof of Lemma \ref{['lemma:informed-direction']}
  • Definition 1: $\alpha$-convexity
  • Definition 2: $\beta$-smoothness
  • proof : Proof of Proposition \ref{['prop:regret']}