Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High dimensional Bayesian Optimization via Condensing-Expansion Projection

Jiaming Lu, Rong J. B. Zhu

TL;DR

This work introduces Condensing-Expansion Projection Bayesian Optimization (CEPBO) to tackle high-dimensional BO without assuming an effective subspace. By repeatedly projecting data to a low-dimensional embedding with a fresh random matrix $\mathbf{A}_t$, performing BO in the embedding, and expanding back to the original space for evaluation, CEPBO preserves information from the high-dimensional domain. Two instantiations, CEP-REMBO (Gaussian) and CEP-HeSBO (hashing), outperform state-of-the-art random-embedding methods across synthetic benchmarks and four real-world problems, supported by theoretical results showing isometry in expectation and concentration of the mapped points. The approach offers practical flexibility and competitive scalability, though it leaves open analysis for epsilon-subspace embeddings and extreme-scale dimensions.

Abstract

In high-dimensional settings, Bayesian optimization (BO) can be expensive and infeasible. The random embedding Bayesian optimization algorithm is commonly used to address high-dimensional BO challenges. However, this method relies on the effective subspace assumption on the optimization problem's objective function, which limits its applicability. In this paper, we introduce Condensing-Expansion Projection Bayesian optimization (CEPBO), a novel random projection-based approach for high-dimensional BO that does not reply on the effective subspace assumption. The approach is both simple to implement and highly practical. We present two algorithms based on different random projection matrices: the Gaussian projection matrix and the hashing projection matrix. Experimental results demonstrate that both algorithms outperform existing random embedding-based algorithms in most cases, achieving superior performance on high-dimensional BO problems. The code is available in \url{https://anonymous.4open.science/r/CEPBO-14429}.

High dimensional Bayesian Optimization via Condensing-Expansion Projection

TL;DR

This work introduces Condensing-Expansion Projection Bayesian Optimization (CEPBO) to tackle high-dimensional BO without assuming an effective subspace. By repeatedly projecting data to a low-dimensional embedding with a fresh random matrix , performing BO in the embedding, and expanding back to the original space for evaluation, CEPBO preserves information from the high-dimensional domain. Two instantiations, CEP-REMBO (Gaussian) and CEP-HeSBO (hashing), outperform state-of-the-art random-embedding methods across synthetic benchmarks and four real-world problems, supported by theoretical results showing isometry in expectation and concentration of the mapped points. The approach offers practical flexibility and competitive scalability, though it leaves open analysis for epsilon-subspace embeddings and extreme-scale dimensions.

Abstract

In high-dimensional settings, Bayesian optimization (BO) can be expensive and infeasible. The random embedding Bayesian optimization algorithm is commonly used to address high-dimensional BO challenges. However, this method relies on the effective subspace assumption on the optimization problem's objective function, which limits its applicability. In this paper, we introduce Condensing-Expansion Projection Bayesian optimization (CEPBO), a novel random projection-based approach for high-dimensional BO that does not reply on the effective subspace assumption. The approach is both simple to implement and highly practical. We present two algorithms based on different random projection matrices: the Gaussian projection matrix and the hashing projection matrix. Experimental results demonstrate that both algorithms outperform existing random embedding-based algorithms in most cases, achieving superior performance on high-dimensional BO problems. The code is available in \url{https://anonymous.4open.science/r/CEPBO-14429}.
Paper Structure (23 sections, 4 theorems, 42 equations, 4 figures, 1 algorithm)

This paper contains 23 sections, 4 theorems, 42 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Bayesian Optimization
  5. Condensing-Expansion Projection
  6. The CEPBO Algorithm
  7. Address the boundary issue
  8. Experiments
  9. Numerical Results
  10. Real-World Problems
  11. Conclusion
  12. Appendix
  13. Proof of Theorem \ref{['expectation_properties']}
  14. Proof of Theorem \ref{['prop:Variance']}
  15. Two Lemmas
  16. ...and 8 more sections

Key Result

Theorem 1

Assume a matrix $\mathbf{A}$ satisfying Definition random Gaussian matrix definition or Definition hashing random projection matrix, it can be shown that

Figures (4)

  • Figure 1: An illustration of CEP. The dimension of the original space is $D=3$, and the dimension of the embedding subspace is $d=2$. The five points in the orginal space is projected to the embedding subspace by Condensing Projection, then they are projected back to the original space by Expansion Projection. The optimal point (red dot) in the original space is still at the (approximately) optimal position after CEP.
  • Figure 2: The results of the optimization experiments for three functions across various embedding dimensions. From top to bottom: Holder Table, Schwefel, and Griewank functions.
  • Figure 3: The results of the optimization experiments for four real-world scenarios. From left to right: Lunar landing, Robot pushing, NAS, and Rover planning.
  • Figure 4: The results of the optimization experiments for Hartmann function.

Theorems & Definitions (10)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Theorem 2
  • proof
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof