Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes

Yan Zhang, Xuefeng Liu, Sipeng Chen, Sascha Ranftl, Chong Liu, Shibo Li

TL;DR

RAMBO addresses the challenge of multi-regime landscapes in Bayesian Optimization by replacing a single stationary GP with a Dirichlet Process Mixture of Gaussian Processes (DPMM-GP). It derives a collapsed Gibbs inference procedure that marginalizes latent functions, and introduces an adaptive concentration parameter schedule to reveal regimes progressively as data accumulates. The acquisition function is a mixture-aware Expected Improvement that decomposes uncertainty into intra-regime variance and inter-regime disagreement, enabling regime-boundary exploration. Empirical results across synthetic benchmarks and three scientific design domains (molecular conformer optimization, drug discovery, and fusion reactor design) show consistent improvements over state-of-the-art baselines in multi-regime objectives, with RAMBO effectively discovering 3–5 regimes in high-dimensional settings.

Abstract

Standard Bayesian Optimization (BO) assumes uniform smoothness across the search space an assumption violated in multi-regime problems such as molecular conformation search through distinct energy basins or drug discovery across heterogeneous molecular scaffolds. A single GP either oversmooths sharp transitions or hallucinates noise in smooth regions, yielding miscalibrated uncertainty. We propose RAMBO, a Dirichlet Process Mixture of Gaussian Processes that automatically discovers latent regimes during optimization, each modeled by an independent GP with locally-optimized hyperparameters. We derive collapsed Gibbs sampling that analytically marginalizes latent functions for efficient inference, and introduce adaptive concentration parameter scheduling for coarse-to-fine regime discovery. Our acquisition functions decompose uncertainty into intra-regime and inter-regime components. Experiments on synthetic benchmarks and real-world applications, including molecular conformer optimization, virtual screening for drug discovery, and fusion reactor design, demonstrate consistent improvements over state-of-the-art baselines on multi-regime objectives.

Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes

TL;DR

RAMBO addresses the challenge of multi-regime landscapes in Bayesian Optimization by replacing a single stationary GP with a Dirichlet Process Mixture of Gaussian Processes (DPMM-GP). It derives a collapsed Gibbs inference procedure that marginalizes latent functions, and introduces an adaptive concentration parameter schedule to reveal regimes progressively as data accumulates. The acquisition function is a mixture-aware Expected Improvement that decomposes uncertainty into intra-regime variance and inter-regime disagreement, enabling regime-boundary exploration. Empirical results across synthetic benchmarks and three scientific design domains (molecular conformer optimization, drug discovery, and fusion reactor design) show consistent improvements over state-of-the-art baselines in multi-regime objectives, with RAMBO effectively discovering 3–5 regimes in high-dimensional settings.

Abstract

Standard Bayesian Optimization (BO) assumes uniform smoothness across the search space an assumption violated in multi-regime problems such as molecular conformation search through distinct energy basins or drug discovery across heterogeneous molecular scaffolds. A single GP either oversmooths sharp transitions or hallucinates noise in smooth regions, yielding miscalibrated uncertainty. We propose RAMBO, a Dirichlet Process Mixture of Gaussian Processes that automatically discovers latent regimes during optimization, each modeled by an independent GP with locally-optimized hyperparameters. We derive collapsed Gibbs sampling that analytically marginalizes latent functions for efficient inference, and introduce adaptive concentration parameter scheduling for coarse-to-fine regime discovery. Our acquisition functions decompose uncertainty into intra-regime and inter-regime components. Experiments on synthetic benchmarks and real-world applications, including molecular conformer optimization, virtual screening for drug discovery, and fusion reactor design, demonstrate consistent improvements over state-of-the-art baselines on multi-regime objectives.
Paper Structure (49 sections, 9 theorems, 41 equations, 4 figures, 2 algorithms)

This paper contains 49 sections, 9 theorems, 41 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Gaussian Processes.
  4. Bayesian Optimization.
  5. Dirichlet Process and Mixture Models
  6. Probabilistic Surrogate for RAMBO
  7. Generative Process of DPMM
  8. Joint Distribution and Marginal Likelihood
  9. Posterior Inference via Collapsed Gibbs Sampling
  10. Sampling Regime Assignments
  11. Updating Hyperparameters
  12. Posterior Predictive Distribution
  13. Acquisition Functions
  14. Mixture Expected Improvement
  15. Adaptive Concentration Parameter Scheduling
  16. ...and 34 more sections

Key Result

Theorem 2

A random measure $G \sim \mathrm{DP}(\alpha, G_0)$ admits the almost sure representation: where the atoms $\theta_k \stackrel{i.i.d.}{\sim} G_0$ and the weights $\{\pi_k\}$ are generated via: $\beta_k \sim \mathrm{Beta}(1, \alpha); \pi_k = \beta_k \prod_{j=1}^{k-1}(1-\beta_j).$

Figures (4)

  • Figure 1: Optimization performance across synthetic and real-world benchmarks. We report the best objective value found (mean $\pm$ SE over 5 seeds). (a)--(d) Levy and Schwefel functions in 6D and 10D. (e)--(g) Molecular conformer optimization (12D), virtual screening for drug discovery (50D), and stellarator reactor design (81D). RAMBO consistently matches or outperforms all baselines, with the largest gains on high-dimensional, multi-regime landscapes.
  • Figure 2: 3D landscape of the 2D Levy function, illustrating the dense clusters of local minima and rugged surface topology.
  • Figure 3: 3D landscape of the 2D Schwefel function, showcasing the deceptive local optima and the isolated nature of the global minimum.
  • Figure 4: Dihedral (Torsion) Angle Diagram: The optimization space consists of 12 such rotational degrees of freedom, where steric hindrance between atoms $1$ and $4$ creates complex energy barriers.

Theorems & Definitions (11)

  • Definition 1: Dirichlet Process
  • Theorem 2: Sethuraman's Stick-Breaking Construction sethuraman1994constructive
  • Theorem 3: Expected Number of Clusters
  • Definition 4: Chinese Restaurant Process
  • Theorem 5: DPMM-GP Joint Distribution
  • Proposition 6: Cluster Marginal Likelihood
  • Theorem 7: Posterior Predictive Density
  • Theorem 8: Moment Matching
  • Theorem 9: DPMM-GP Expected Improvement
  • Theorem 10: DPMM-GP Probability of Improvement
  • ...and 1 more