Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces

Leonard Papenmeier, Luigi Nardi, Matthias Poloczek

TL;DR

The paper tackles high-dimensional, mixed- and combinatorial-space black-box optimization by introducing Bounce, a Bayesian optimization framework that learns through nested random embeddings into increasingly high-dimensional subspaces. Bounce combines a Count-Sketch–based subspace embedding, dynamic trust-region management, and batch acquisition with the CoCaBo kernel to model correlations across continuous and combinatorial variables, enabling scalable, parallel evaluations. Empirical results across multiple challenging benchmarks (LABS, ClusterExpansion, Pest Control, SVM, etc.) show Bounce is on par with or superior to state-of-the-art methods and demonstrates robustness to the location of the unknown optimum, addressing reliability concerns in existing approaches. The work provides a practical, open-source solution that broadens the applicability of BO in real-world, high-dimensional mixed spaces, with potential impact in materials discovery, hardware design, and automated ML tasks.

Abstract

Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces. While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. Their performances degrade substantially when the unknown optima of the function do not have a certain structure. To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality. Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.

Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces

TL;DR

The paper tackles high-dimensional, mixed- and combinatorial-space black-box optimization by introducing Bounce, a Bayesian optimization framework that learns through nested random embeddings into increasingly high-dimensional subspaces. Bounce combines a Count-Sketch–based subspace embedding, dynamic trust-region management, and batch acquisition with the CoCaBo kernel to model correlations across continuous and combinatorial variables, enabling scalable, parallel evaluations. Empirical results across multiple challenging benchmarks (LABS, ClusterExpansion, Pest Control, SVM, etc.) show Bounce is on par with or superior to state-of-the-art methods and demonstrates robustness to the location of the unknown optimum, addressing reliability concerns in existing approaches. The work provides a practical, open-source solution that broadens the applicability of BO in real-world, high-dimensional mixed spaces, with potential impact in materials discovery, hardware design, and automated ML tasks.

Abstract

Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces. While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. Their performances degrade substantially when the unknown optima of the function do not have a certain structure. To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality. Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.
Paper Structure (51 sections, 1 theorem, 10 equations, 22 figures, 1 algorithm)

This paper contains 51 sections, 1 theorem, 10 equations, 22 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Bayesian optimization.
  4. Combinatorial domains.
  5. High-dimensional continuous spaces.
  6. Combinatorial high-dimensional domains.
  7. The Bounce algorithm
  8. The nested subspaces.
  9. Trust region management.
  10. Batch parallelism.
  11. The kernel choice.
  12. The subspace embedding of mixed spaces
  13. Continuous variables.
  14. Binary variables.
  15. Categorical and ordinal variables.
  16. ...and 36 more sections

Key Result

Theorem 1

With the following definitions and under the following assumptions: then the Bounce algorithm finds a global optimum with probability 1, as the number of samples $N$ goes to $\infty$.

Figures (22)

  • Figure 1: The mapping (or binning) of categorical and ordinal variables. Suppose that variable $v_k$ has two categories and that $v_{\ell}$ has three categories. Both are mapped to the target dimension $d_i$ that has cardinality $3 = \max\{2,3\}$. While the mapping of $v_{\ell}$ to $d_i$ is a straightforward bijection, $v_k$ has fewer categories than $d_i$. Thus, the mapping of $v_k$ to $d_i$ repeats label 1. Ordinal variables are mapped similarly.
  • Figure 2: The 50D low-autocorrelation binary sequence problem. Bounce finds the best solutions, followed by COMBO.
  • Figure 3: The 125D weighted ClusterExpansion maximum satisfiability problem. We plot the total negative weight of clauses. Bounce produces the best assignments.
  • Figure 4: The 25D categorical pest control problem. Bounce obtains the best solutions, followed by Casmopolitan. BODi's performance degrades significantly when shuffling the order of categories.
  • Figure 5: The 53-dimensional SVM benchmark. Bounce, BODi, and Casmopolitan achieve comparable solutions.
  • ...and 17 more figures

Theorems & Definitions (2)

  • Theorem 1: Bounce consistency
  • proof