Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization
Zhitong Xu, Haitao Wang, Jeff M Phillips, Shandian Zhe
TL;DR
This study re-evaluates standard Bayesian Optimization with Gaussian processes in high-dimensional settings, identifying gradient vanishing as the primary failure mode when using SE kernels with typical length-scale initializations. It provides a theoretical characterization comparing SE and Matérn kernels and introduces a simple robust initialization ell0 = c√d that exponentially reduces gradient vanishing, enabling SE-based BO to perform competitively. Empirically, standard BO with Matérn kernels or with the proposed robust initialization achieves top-tier performance across 12 benchmarks (30–1003 dimensions), often matching or surpassing specialized high-dimensional BO methods. The results advocate reconsidering standard GP-based BO's capabilities in high dimensions and offer a practical, priors-free initialization to mitigate training difficulties.
Abstract
A long-standing belief holds that Bayesian Optimization (BO) with standard Gaussian processes (GP) -- referred to as standard BO -- underperforms in high-dimensional optimization problems. While this belief seems plausible, it lacks both robust empirical evidence and theoretical justification. To address this gap, we present a systematic investigation. First, through a comprehensive evaluation across twelve benchmarks, we found that while the popular Square Exponential (SE) kernel often leads to poor performance, using Matérn kernels enables standard BO to consistently achieve top-tier results, frequently surpassing methods specifically designed for high-dimensional optimization. Second, our theoretical analysis reveals that the SE kernel's failure primarily stems from improper initialization of the length-scale parameters, which are commonly used in practice but can cause gradient vanishing in training. We provide a probabilistic bound to characterize this issue, showing that Matérn kernels are less susceptible and can robustly handle much higher dimensions. Third, we propose a simple robust initialization strategy that dramatically improves the performance of the SE kernel, bringing it close to state-of-the-art methods, without requiring additional priors or regularization. We prove another probabilistic bound that demonstrates how the gradient vanishing issue can be effectively mitigated with our method. Our findings advocate for a re-evaluation of standard BO's potential in high-dimensional settings.