Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization

Yuejiang Wen, Paul D. Franzon

TL;DR

This work tackles high-dimensional Bayesian optimization by extending REMBO to SA-REMBO, which jointly models multiple random embeddings via an embedding-indexed Gaussian Process with a product kernel. Theoretical results (Theorems 1 and 2) establish that low-dimensional projections can preserve the essential subspace structure needed to recover the global optimum, enabling robust optimization across multiple subspaces. Empirically, SA-REMBO demonstrates superior performance and robustness on nonstationary, multimodal benchmarks such as Styblinski–Tang and Harzmann6, outperforming traditional REMBO and similar low-rank BO approaches. The approach offers a scalable, flexible framework for structured design spaces where local subspace structure varies across the domain.

Abstract

Bayesian Optimization (BO) in high-dimensional spaces remains fundamentally limited by the curse of dimensionality and the rigidity of global low-dimensional assumptions. While Random EMbedding Bayesian Optimization (REMBO) mitigates this via linear projections into low-dimensional subspaces, it typically assumes a single global embedding and a stationary objective. In this work, we introduce Self-Adaptive embedding REMBO (SA-REMBO), a novel framework that generalizes REMBO to support multiple random Gaussian embeddings, each capturing a different local subspace structure of the high-dimensional objective. An index variable governs the embedding choice and is jointly modeled with the latent optimization variable via a product kernel in a Gaussian Process surrogate. This enables the optimizer to adaptively select embeddings conditioned on location, effectively capturing locally varying effective dimensionality, nonstationarity, and heteroscedasticity in the objective landscape. We theoretically analyze the expressiveness and stability of the index-conditioned product kernel and empirically demonstrate the advantage of our method across synthetic and real-world high-dimensional benchmarks, where traditional REMBO and other low-rank BO methods fail. Our results establish SA-REMBO as a powerful and flexible extension for scalable BO in complex, structured design spaces.

Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization

TL;DR

This work tackles high-dimensional Bayesian optimization by extending REMBO to SA-REMBO, which jointly models multiple random embeddings via an embedding-indexed Gaussian Process with a product kernel. Theoretical results (Theorems 1 and 2) establish that low-dimensional projections can preserve the essential subspace structure needed to recover the global optimum, enabling robust optimization across multiple subspaces. Empirically, SA-REMBO demonstrates superior performance and robustness on nonstationary, multimodal benchmarks such as Styblinski–Tang and Harzmann6, outperforming traditional REMBO and similar low-rank BO approaches. The approach offers a scalable, flexible framework for structured design spaces where local subspace structure varies across the domain.

Abstract

Bayesian Optimization (BO) in high-dimensional spaces remains fundamentally limited by the curse of dimensionality and the rigidity of global low-dimensional assumptions. While Random EMbedding Bayesian Optimization (REMBO) mitigates this via linear projections into low-dimensional subspaces, it typically assumes a single global embedding and a stationary objective. In this work, we introduce Self-Adaptive embedding REMBO (SA-REMBO), a novel framework that generalizes REMBO to support multiple random Gaussian embeddings, each capturing a different local subspace structure of the high-dimensional objective. An index variable governs the embedding choice and is jointly modeled with the latent optimization variable via a product kernel in a Gaussian Process surrogate. This enables the optimizer to adaptively select embeddings conditioned on location, effectively capturing locally varying effective dimensionality, nonstationarity, and heteroscedasticity in the objective landscape. We theoretically analyze the expressiveness and stability of the index-conditioned product kernel and empirically demonstrate the advantage of our method across synthetic and real-world high-dimensional benchmarks, where traditional REMBO and other low-rank BO methods fail. Our results establish SA-REMBO as a powerful and flexible extension for scalable BO in complex, structured design spaces.
Paper Structure (19 sections, 20 equations, 13 figures, 1 algorithm)

This paper contains 19 sections, 20 equations, 13 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. REMBO
  3. Theoretical Foundations of REMBO
  4. Theorem 1: Effective Dimensionality and Decomposition
  5. Theorem 2: Rank Preservation and Random Projections
  6. Practical Implications of These Theorems
  7. Related Work
  8. Method Description
  9. Kernel Construction
  10. Acquisition Optimization
  11. Theoretical Implications
  12. Advantages and Limitations
  13. Advantages
  14. Limitations
  15. Experiments
  16. ...and 4 more sections

Figures (13)

  • Figure 1: Rndom Cross Embedding used by cREMBO in 2D
  • Figure 4: The Automatic Choice of Embedding in cREMBO.
  • Figure 5: The Optimization Results Comparisons of REMBO and cREMBO.
  • Figure : (a) Random Embedding
  • Figure : (a) Styblinski–Tang Function (d=8 D =21)
  • ...and 8 more figures