Stein Boltzmann Sampling: A Variational Approach for Global Optimization
Gaëtan Serré, Argyris Kalogeratos, Nicolas Vayatis
TL;DR
The paper introduces Stein Boltzmann Sampling (SBS), a flow-based approach for global optimization of continuous Sobolev functions that uses Stein Variational Gradient Descent to move an initial uniform particle set toward a Boltzmann target $m^{(\kappa)}(x) \propto e^{-\kappa f(x)}$ on a compact domain $\Omega$. By extending SVGD theory to BD targets on $\Omega$, the authors prove weak convergence of the particle flow to the target distribution and establish SBS as asymptotically convergent to the global minimum as $\kappa$, the number of particles $N$, and the step size $\varepsilon$ grow appropriately; they also relate the KS discrepancy to the optimization objective. They present two practical variants, SBS-PF and SBS-HYBRID, to improve budget efficiency and to combine SBS with other optimization methods, respectively. Empirical results on standard benchmark functions show SBS variants outperform many state-of-the-art methods in average performance and offer favorable accuracy-budget trade-offs, with SBS-PF achieving substantial budget reductions and SBS-HYBRID often delivering top performance in practice.
Abstract
In this paper, we present a flow-based method for global optimization of continuous Sobolev functions, called Stein Boltzmann Sampling (SBS). SBS initializes uniformly a number of particles representing candidate solutions, then uses the Stein Variational Gradient Descent (SVGD) algorithm to sequentially and deterministically move those particles in order to approximate a target distribution whose mass is concentrated around promising areas of the domain of the optimized function. The target is chosen to be a properly parametrized Boltzmann distribution. For the purpose of global optimization, we adapt the generic SVGD theoretical framework allowing to address more general target distributions over a compact subset of $\mathbb{R}^d$, and we prove SBS's asymptotic convergence. In addition to the main SBS algorithm, we present two variants: the SBS-PF that includes a particle filtering strategy, and the SBS-HYBRID one that uses SBS or SBS-PF as a continuation after other particle- or distribution-based optimization methods. A detailed comparison with state-of-the-art methods on benchmark functions demonstrates that SBS and its variants are highly competitive, while the combination of the two variants provides the best trade-off between accuracy and computational cost.