High-Dimensional Bayesian Optimization via Random Projection of Manifold Subspaces

TL;DR

The paper tackles the curse of dimensionality in Bayesian optimization by proposing RPM‑BO, a method that unifies random linear projections with learned latent manifold representations. By expressing the objective as $f=g\circ h$ and decomposing $g$ into a random projection $oldsymbol{A}$ and a low‑dimensional regressor, the acquisition function is optimized in a reduced space and then back‑projected to the original high‑dimensional domain. The authors introduce geometry‑aware and geometry‑unaware mappings for $h$, along with a semi‑supervised training objective that mitigates overfitting, and show that random projections preserve the manifold geometry with high probability, enabling convergence guarantees and efficient optimization. Empirically, RPM‑BO outperforms several HD‑BO baselines on synthetic benchmarks with spherical and mixed manifolds and on real‑world tasks such as LassoBench and MuJoColocomotion, demonstrating scalable performance in dimensions up to 1500+.

Abstract

Bayesian Optimization (BO) is a popular approach to optimizing expensive-to-evaluate black-box functions. Despite the success of BO, its performance may decrease exponentially as the dimensionality increases. A common framework to tackle this problem is to assume that the objective function depends on a limited set of features that lie on a low-dimensional manifold embedded in the high-dimensional ambient space. The latent space can be linear or more generally nonlinear. To learn feature mapping, existing works usually use an encode-decoder framework which is either computationally expensive or susceptible to overfittting when the labeled data is limited. This paper proposes a new approach for BO in high dimensions by exploiting a new representation of the objective function. Our approach combines a random linear projection to reduce the dimensionality, with a representation learning of the nonlinear manifold. When the geometry of the latent manifold is available, a solution to exploit this geometry is proposed for representation learning. In contrast, we use a neural network. To mitigate overfitting by using the neural network, we train the feature mapping in a geometry-aware semi-supervised manner. Our approach enables efficient optimizing of BO's acquisition function in the low-dimensional space, with the advantage of projecting back to the original high-dimensional space compared to existing works in the same setting. Finally, we show empirically that our algorithm outperforms other high-dimensional BO baselines in various synthetic functions and real applications.

Figures (10)

• Figure 1: Left: The spherical $2-$dimensional manifold $\mathcal{S}^2$ embedded in $\mathbb{R}^3$. Right: The mixed $2-$dimensional manifold $\mathbf{M}^2$ embedded in $\mathbb{R}^3$ with $(x,y) \in \mathcal{T}^1$ and $z \in R$ .
• Figure 2: Performances on two standard functions with effective spherical manifold for 500, 1000, and 1500 input dimensions. For all cases, the dimension of the effective manifold is 10. The $y-$axis presents the value function (A smaller value is better).
• Figure 3: Performances on two standard functions with effective mixed-manifold for 500, 1000, and 1500 input dimensions. For all cases, the dimension of the effective manifold is 15. The $y-$axis presents the value function (A smaller value is better).
• Figure 4: Performance on three real applications.
• Figure 5: Performance on Ackley function with effective 15-dimensional mix manifold $\mathbf M$ for varying projection dimension $m$.

Theorems & Definitions (11)

• proposition thmcounterproposition
• theorem thmcountertheorem
• proof
• theorem thmcountertheorem
• proof
• lemma thmcounterlemma
• proof
• proof
• definition thmcounterdefinition
• theorem thmcountertheorem