Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating in High-Dimensional Search Space: A Hierarchical Bayesian Optimization Approach

Wenxuan Li, Taiyi Wang, Eiko Yoneki

TL;DR

HiBO presents a hierarchical Bayesian optimization framework that integrates a data-driven, adaptive global search-space partitioning navigator with a local BO optimizer to tackle high-dimensional black-box optimization. By constructing a partition tree and evaluating sampling potential with UCT, HiBO informs a partition-weighted acquisition that balances exploration and exploitation across the whole search space. Theoretical results show exponential pruning of sub-optimal regions under small partition errors, and extensive experiments demonstrate HiBO’s superior performance on synthetic high-dimensional benchmarks and a real-world DBMS tuning task, with favorable safety and efficiency metrics. The work offers a scalable approach to high-dimensional BO with practical implications for complex system configuration and automated tuning tasks.

Abstract

Optimizing black-box functions in high-dimensional search spaces has been known to be challenging for traditional Bayesian Optimization (BO). In this paper, we introduce HiBO, a novel hierarchical algorithm integrating global-level search space partitioning information into the acquisition strategy of a local BO-based optimizer. HiBO employs a search-tree-based global-level navigator to adaptively split the search space into partitions with different sampling potential. The local optimizer then utilizes this global-level information to guide its acquisition strategy towards most promising regions within the search space. A comprehensive set of evaluations demonstrates that HiBO outperforms state-of-the-art methods in high-dimensional synthetic benchmarks and presents significant practical effectiveness in the real-world task of tuning configurations of database management systems (DBMSs).

Navigating in High-Dimensional Search Space: A Hierarchical Bayesian Optimization Approach

TL;DR

HiBO presents a hierarchical Bayesian optimization framework that integrates a data-driven, adaptive global search-space partitioning navigator with a local BO optimizer to tackle high-dimensional black-box optimization. By constructing a partition tree and evaluating sampling potential with UCT, HiBO informs a partition-weighted acquisition that balances exploration and exploitation across the whole search space. Theoretical results show exponential pruning of sub-optimal regions under small partition errors, and extensive experiments demonstrate HiBO’s superior performance on synthetic high-dimensional benchmarks and a real-world DBMS tuning task, with favorable safety and efficiency metrics. The work offers a scalable approach to high-dimensional BO with practical implications for complex system configuration and automated tuning tasks.

Abstract

Optimizing black-box functions in high-dimensional search spaces has been known to be challenging for traditional Bayesian Optimization (BO). In this paper, we introduce HiBO, a novel hierarchical algorithm integrating global-level search space partitioning information into the acquisition strategy of a local BO-based optimizer. HiBO employs a search-tree-based global-level navigator to adaptively split the search space into partitions with different sampling potential. The local optimizer then utilizes this global-level information to guide its acquisition strategy towards most promising regions within the search space. A comprehensive set of evaluations demonstrates that HiBO outperforms state-of-the-art methods in high-dimensional synthetic benchmarks and presents significant practical effectiveness in the real-world task of tuning configurations of database management systems (DBMSs).

Paper Structure

This paper contains 41 sections, 1 theorem, 14 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Preliminaries
  4. Related Work
  5. Methodology: Hierarchical Bayesian Optimization
  6. Overview
  7. Global-level Navigator: Adaptive Search Space Partitioning
  8. Data-Driven Search-Tree-based Space Partitioning
  9. Sampling Potential Evaluation via UCT
  10. Dynamic and Adaptive Search Tree Construction
  11. Local Optimizer: Partition-Score-Weighted Acquisition
  12. Theoretical Analysis
  13. Bounding the Most Promising Partition
  14. Implications
  15. Evaluation
  16. ...and 26 more sections

Key Result

Theorem 4.1

(Bound on Most Promising Leaf). Consider a partition tree of height $h$ built by recursively splitting the sub-domain with the highest partition score. Let $\delta_{\max}$ be the maximum partition error of any node along that "leftmost" path. Then the sub-domain (leaf) at depth $h$ on that path cont

Figures (9)

  • Figure 1: Illustration of the high-level workflow of HiBO within each optimization iteration.
  • Figure 2: Illustration of how the constructed search tree is integrated into the local optimizer's acquisition strategy.
  • Figure 3: Evaluation results of algorithms on selected synthetic benchmarks.
  • Figure 4: Left: Performance (throughput) evaluation of PostgreSQL being tuned by different algorithms on SysBench. Note that the figure for SysBench omits the low-performing part (throughput < 250 EPS) for readability. Right: S-PITR measurements of experiments done with the selected algorithms, which is explained with details in Section \ref{['sec:db-res']}.
  • Figure 5: Visualization of distribution of partition-score-weighted acquisition scores (HiBO) and vanilla acquisition scores (TuRBO) on the first two dimensions of Ackley2-200D across iterations. Lighter color indicates greater acquisition values and the green 'X' represents the optimal point (0, 0).
  • ...and 4 more figures

Theorems & Definitions (3)

  • Theorem 4.1
  • proof
  • proof