Stacking Variational Bayesian Monte Carlo

TL;DR

This work addresses the challenge of approximating complex, expensive posteriors by VBMC, which can miss global structure due to its local exploration. The authors introduce Stacking VBMC (S-VBMC), a post-processing method that merges multiple independent VBMC runs into a single global posterior by reweighting components in a shared space, without additional likelihood evaluations. Across synthetic benchmarks with multimodality and narrow tails and two real neuroscience problems, S-VBMC consistently improves posterior accuracy (as measured by MMTV and GsKL) and remains computationally efficient when runs are executed in parallel. A practical debiasing heuristic mitigates an ELBO overestimation bias that can arise with noisy log-likelihoods, enhancing reliability for model comparison, while maintaining a simple, robust, and easily integrable workflow.

Abstract

Approximate Bayesian inference for models with computationally expensive, black-box likelihoods poses a significant challenge, especially when the posterior distribution is complex. Many inference methods struggle to explore the parameter space efficiently under a limited budget of likelihood evaluations. Variational Bayesian Monte Carlo (VBMC) is a sample-efficient method that addresses this by building a local surrogate model of the log-posterior. However, its conservative exploration strategy, while promoting stability, can cause it to miss important regions of the posterior, such as distinct modes or long tails. In this work, we introduce Stacking Variational Bayesian Monte Carlo (S-VBMC), a method that overcomes this limitation by constructing a robust, global posterior approximation from multiple independent VBMC runs. Our approach merges these local approximations through a principled and inexpensive post-processing step that leverages VBMC's mixture posterior representation and per-component evidence estimates. Crucially, S-VBMC requires no additional likelihood evaluations and is naturally parallelisable, fitting seamlessly into existing inference workflows. We demonstrate its effectiveness on two synthetic problems designed to challenge VBMC's exploration and two real-world applications from computational neuroscience, showing substantial improvements in posterior approximation quality across all cases. Our code is available as a Python package at https://github.com/acerbilab/svbmc.

Figures (17)

• Figure 1: Two separate VBMC posteriors (left, shown as blue and orange contours) and the resulting stacked posterior via S-VBMC (right, red contour) for a neuronal model with real data (see Section \ref{['sec:rw']}); showing the marginal distribution of two out of the five model parameters.
• Figure 2: Examples of overlap between the ground truth and the posterior when combining different numbers of VBMC runs on the GMM (top panels) and ring (bottom panels) synthetic benchmarks. The red points indicate samples from the posterior approximation, with the target density depicted with colour gradients in the background.
• Figure 3: Synthetic problems. Metrics plotted as a function of the number of VBMC runs stacked (median and 95% confidence interval). Metrics are plotted for S-VBMC (blue), VBMC (red), and NS (grey). The best BBVI results are shown in green. The black horizontal line in the ELBO panels represents the ground-truth LML, while the dashed lines on $\Delta$LML, MMTV, and GsKL denote desirable thresholds for each metric (good performance is below the threshold; see Section \ref{['sec:metrics']})
• Figure 4: Real-world problems. Metrics plotted as a function of the number of VBMC runs stacked (median and 95% confidence interval). Metrics are plotted for S-VBMC (blue), VBMC (red), and NS (grey). The best BBVI results are shown in green. The black horizontal line in the ELBO panels represents the ground-truth LML, while the dashed lines on $\Delta$LML, MMTV, and GsKL denote desirable thresholds for each metric (good performance is below the threshold; see Section \ref{['sec:metrics']}). The BBVI error bar in the plot displaying the ELBO in the neuronal model (top left) is truncated for clarity.
• Figure 5: Compute time of a single VBMC run (red) and post-processing time only (i.e., computational overhead) of S-VBMC (blue) plotted as a function of the number of VBMC runs stacked (median and 95% confidence interval, computed from 10000 bootstrap resamples). Each subplot represents a different benchmark problem. The values plotted here correspond to the actual computation times of the experiments described in Sections \ref{['sec:synthetic']} and \ref{['sec:rw']}.