Bayesian Optimization by Kernel Regression and Density-based Exploration

TL;DR

This work tackles the computational bottleneck of Gaussian-process Bayesian optimization by introducing BOKE, which replaces the GP surrogate with kernel regression and augments it with kernel-density exploration. The IKR-UCB acquisition balances exploitation and exploration, and the authors prove pointwise consistency, algorithmic consistency, and regret bounds, while reporting a total complexity that scales as $O(mT^2)$ rather than the GP-era $O(T^4)$. Theoretical results ensure dense sampling of $\mathcal{X}$ and convergence of simple regret, complemented by empirical results on synthetic benchmarks, a sprinkler simulation, and hyperparameter tuning that demonstrate competitive performance with substantial computational savings. The method offers a practical, robust alternative for optimization in resource-constrained settings and opens avenues for refined analyses and extensions to other surrogate models.

Abstract

Bayesian optimization is highly effective for optimizing expensive-to-evaluate black-box functions, but it faces significant computational challenges due to the high computational complexity of Gaussian processes, which results in a total time complexity that is quartic with respect to the number of iterations. To address this limitation, we propose the Bayesian Optimization by Kernel regression and density-based Exploration (BOKE) algorithm. BOKE uses kernel regression for efficient function approximation, kernel density for exploration, and integrates them into the confidence bound criteria to guide the optimization process, thus reducing computational costs to quadratic. Our theoretical analysis rigorously establishes the global convergence of BOKE and ensures its robustness in noisy settings. Through extensive numerical experiments on both synthetic and real-world optimization tasks, we demonstrate that BOKE not only performs competitively compared to Gaussian process-based methods and several other baseline methods but also exhibits superior computational efficiency. These results highlight BOKE's effectiveness in resource-constrained environments, providing a practical approach for optimization problems in engineering applications.

Figures (9)

• Figure 1: Kernel regression with Gaussian kernels at various bandwidths. The objective function Eq. \ref{['eqn:test']} is sampled $10$ times, with each sample evaluated incorporating additive noise $\varepsilon \sim \mathcal{N}(0, 0.01)$.
• Figure 2: Kernel density estimation with Gaussian kernels at various bandwidths, where $\hat{p}(x) = (2 \pi)^{-1/2} (\ell t)^{-1} W_t(x)$. The $20$ samples are generated from $p(x) = 0.7 \, \mathrm{Beta}(3, 8) + 0.3 \, \mathrm{Beta}(10, 2)$.
• Figure 3: Fill distances of different space-filling design methods across various dimensions. Each curve represents the mean result from $100$ random seeds. Both axes are displayed using a logarithmic scale.
• Figure 4: Illustration of the IKR-UCB acquisition function. The gray dashed line represents the analytical function, while the blue curve shows the KR approximation of the objective function. The upper light blue shaded area indicates the uncertainty quantification based on kernel density estimation. The lower shaded plot visualizes the acquisition function.
• Figure 5: Comparison of BOKE (top two rows) and BOKE+ (bottom two rows) on the one-dimensional problem defined in Eq. \ref{['eqn:test']}. The gray dashed line represents the analytical function, while the blue curve shows the KR approximation of the objective function. The shaded plot visualizes the acquisition function.

Theorems & Definitions (34)

• Proposition 3.1
• proof
• Theorem 4.1
• proof
• Lemma 4.2: SNEB property of IKR-UCB
• Theorem 4.3: Algorithmic consistency of BOKE
• proof
• Corollary 4.4: Algorithmic consistency of BOKE+
• proof
• Corollary 4.5: Algorithmic consistency of DE