Stein Variational Evolution Strategies

Abstract

Stein Variational Gradient Descent (SVGD) is a highly efficient method to sample from an unnormalized probability distribution. However, the SVGD update relies on gradients of the log-density, which may not always be available. Existing gradient-free versions of SVGD make use of simple Monte Carlo approximations or gradients from surrogate distributions, both with limitations. To improve gradient-free Stein variational inference, we combine SVGD steps with evolution strategy (ES) updates. Our results demonstrate that the resulting algorithm generates high-quality samples from unnormalized target densities without requiring gradient information. Compared to prior gradient-free SVGD methods, we find that the integration of the ES update in SVGD significantly improves the performance on multiple challenging benchmark problems.

Figures (14)

• Figure 1: Left: Illustration of Stein Variational CMA-ES. Multiple ES search distributions are updated in parallel, similar to the SVGD step. Middle: Quantitative comparison of different methods for sampling and RL control tasks. SV-CMA-ES obtains higher quality solutions than existing gradient-free SVGD-based approaches. Right: Qualitative comparison of different CMA-ES-based methods unveils that SV-CMA-ES generates more diverse samples than other methods. The full experimental details can be found in \ref{['secExp']}.
• Figure 2: Samples obtained by various methods. Gradient-based SVGD (b) captures all target densities effectively, while SV-CMA-ES produces the highest quality samples among gradient-free methods. GF-SVGD struggles on more complex targets, and SV-OpenAI-ES tends to converge slowly due to taking small steps in flat regions of the target.
• Figure 3: (a)-(c): MMD w.r.t. ground truth samples on the synthetic densities depicted in \ref{['fig:samples']}. (d): Mean log10 MMD across all three sampling tasks w.r.t. the samples obtained by gradient-based SVGD. All results are averaged across 10 independent runs ($\pm 1.96$ standard error). SV-CMA-ES approximates the ground truth samples and results by gradient-based SVGD (blue line) the best out of all gradient-free methods.
• Figure 4: Results of Bayesian logistic regression. We report mean ($\pm 1.96$ standard error) across 10 independent runs. SV-CMA-ES converges the faster than other gradient-free methods, and achieves similar performance levels at convergence as gradient-based SVGD (dashed line).
• Figure 5: Results of sampling MLP parameters for RL tasks. Plotted is the best expected return across all particles for each method. We report the mean ($\pm 1.96$ standard error) across 10 independent runs. SV-CMA-ES performs better than the gradient-free baselines across all tasks.