Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regional Expected Improvement for Efficient Trust Region Selection in High-Dimensional Bayesian Optimization

Nobuo Namura, Sho Takemori

TL;DR

This work addresses the challenge of scalable Bayesian optimization in medium to high dimensions by introducing regional expected improvement (REI) and its Monte Carlo variant qREI, integrated into the TuRBO trust-region framework. The authors develop a region-averaging operator that computes acquisition functions over trust regions, and prove that region averaging reduces the problem complexity via a regret bound U_N, with U_N(Sf) ≤ U_N(f). Empirically, REI/qREI-guided trust-region selection improves optimization performance across several real-world high-dimensional tasks, especially when multiple restarts are possible, and demonstrates robustness to unknown problem characteristics. The proposed approach offers a practical, theoretically grounded enhancement for global optimization in high-dimensional, expensive-black-box settings, with publicly available code for replication.

Abstract

Real-world optimization problems often involve complex objective functions with costly evaluations. While Bayesian optimization (BO) with Gaussian processes is effective for these challenges, it suffers in high-dimensional spaces due to performance degradation from limited function evaluations. To overcome this, simplification techniques like dimensionality reduction have been employed, yet they often rely on assumptions about the problem characteristics, potentially underperforming when these assumptions do not hold. Trust-region-based methods, which avoid such assumptions, focus on local search but risk stagnation in local optima. In this study, we propose a novel acquisition function, regional expected improvement (REI), designed to enhance trust-region-based BO in medium to high-dimensional settings. REI identifies regions likely to contain the global optimum, improving performance without relying on specific problem characteristics. We provide a theoretical proof that REI effectively identifies optimal trust regions and empirically demonstrate that incorporating REI into trust-region-based BO outperforms conventional BO and other high-dimensional BO methods in medium to high-dimensional real-world problems.

Regional Expected Improvement for Efficient Trust Region Selection in High-Dimensional Bayesian Optimization

TL;DR

This work addresses the challenge of scalable Bayesian optimization in medium to high dimensions by introducing regional expected improvement (REI) and its Monte Carlo variant qREI, integrated into the TuRBO trust-region framework. The authors develop a region-averaging operator that computes acquisition functions over trust regions, and prove that region averaging reduces the problem complexity via a regret bound U_N, with U_N(Sf) ≤ U_N(f). Empirically, REI/qREI-guided trust-region selection improves optimization performance across several real-world high-dimensional tasks, especially when multiple restarts are possible, and demonstrates robustness to unknown problem characteristics. The proposed approach offers a practical, theoretically grounded enhancement for global optimization in high-dimensional, expensive-black-box settings, with publicly available code for replication.

Abstract

Real-world optimization problems often involve complex objective functions with costly evaluations. While Bayesian optimization (BO) with Gaussian processes is effective for these challenges, it suffers in high-dimensional spaces due to performance degradation from limited function evaluations. To overcome this, simplification techniques like dimensionality reduction have been employed, yet they often rely on assumptions about the problem characteristics, potentially underperforming when these assumptions do not hold. Trust-region-based methods, which avoid such assumptions, focus on local search but risk stagnation in local optima. In this study, we propose a novel acquisition function, regional expected improvement (REI), designed to enhance trust-region-based BO in medium to high-dimensional settings. REI identifies regions likely to contain the global optimum, improving performance without relying on specific problem characteristics. We provide a theoretical proof that REI effectively identifies optimal trust regions and empirically demonstrate that incorporating REI into trust-region-based BO outperforms conventional BO and other high-dimensional BO methods in medium to high-dimensional real-world problems.

Paper Structure

This paper contains 43 sections, 4 theorems, 16 equations, 9 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Gaussian Process
  5. Bayesian Optimization
  6. Trust Region Bayesian Optimization
  7. Proposed Method
  8. Region-Averaged Acquisition Functions
  9. Regional Expected Improvement
  10. Trust Region Selection
  11. Theoretical Analysis
  12. Notation and Assumptions.
  13. Region-Averaging Operator.
  14. Problem Complexity of BO.
  15. Main Theorem.
  16. ...and 28 more sections

Key Result

Theorem 5.1

For any objective function $f \in \mathcal{H}_F(\Omega)$, let $U_N(f)$ be the regret upper bound defined as eq:u_n-def. Then, for any $N \ge 1$, $U_N(f)$ is reduced if we take the operator $\mathcal{S}$, i.e., we have $\mathcal{S} f \in \mathcal{H}_F(\Omega)$ and the following inequality holds:

Figures (9)

  • Figure 1: Comparison of qREI and EI distributions
  • Figure 2: A simple example of the objective function $f$ and its region-averaged function $\mathcal{S} f$.
  • Figure 3: Histories and violin plots of objective function values for the medium-dimensional problems.
  • Figure 4: Histories and violin plots of objective function values for the high-dimensional problems.
  • Figure 5: Critical difference diagram for the medium-dimensional problems
  • ...and 4 more figures

Theorems & Definitions (5)

  • Theorem 5.1
  • Proposition A.1
  • Proposition A.2
  • Proposition A.3
  • proof : Proof of \ref{['thm:reduced-upper-bound']}