Efficient prior sensitivity analysis for Bayesian model comparison

TL;DR

This work tackles the prior sensitivity of Bayesian evidence in model comparison by introducing a post-hoc, sampling-agnostic framework based on the learned harmonic mean estimator (LHME). It reuses existing posterior samples through importance-resampling to evaluate alternative priors, using a learned target density to stabilize the evidence computation and a two-stage diagnostics (Pareto-$\hat{k}$ and fractional ESS) to decide when retraining is needed. The approach reproduces evidences obtained from full re-fitting and nested sampling across toy problems and a cosmological case, achieving up to $\sim 6{,}000\times$ speed-ups. The method enables efficient, transparent assessment of prior influence on model comparison with practical applicability across domains; code will be publicly released in the Harmonic package.

Abstract

Bayesian model comparison implements Occam's razor through its sensitivity to the prior. However, prior-dependence makes it important to assess the influence of plausible alternative priors. Such prior sensitivity analyses for the Bayesian evidence are expensive, either requiring repeated, costly model re-fits or specialised sampling schemes. By exploiting the learned harmonic mean estimator (LHME) for evidence calculation we decouple sampling and evidence calculation, allowing resampled posterior draws to be used directly to calculate the evidence without further likelihood evaluations. This provides an alternative approach to prior sensitivity analysis for Bayesian model comparison that dramatically alleviates the computational cost and is agnostic to the method used to generate posterior samples. We validate our method on toy problems and a cosmological case study, reproducing estimates obtained by full Markov chain Monte Carlo (MCMC) sampling and nested sampling re-fits. For the cosmological example considered our approach achieves up to $6000\times$ lower computational cost.

Figures (4)

• Figure 1: Schematic of the prior sensitivity workflow. We first resample existing posterior draws to the posterior under the new prior. We then perform a two-stage diagnostic check: 1. if the Pareto-$\hat{k}$ diagnostic exceeds 0.7, the importance weights are unreliable and a full model re-fit is required; 2. if $\hat{k}$ is safe but fractional ESS is low ($<0.95$), we re-train the learned target $\varphi(\theta)$ to maintain low variance (blue). After the two checks, the evidence for the alternative prior is computed by Eq. \ref{['eq:reweighted_evidence']}.
• Figure 2: (a) Illustration of prior widths versus the likelihood for the Gaussian toy example, in terms of the 1D marginal density (log-scale). (b) Rosenbrock posterior contours under an uninformative uniform prior (navy) and contours for alternative informative Gaussian priors (red) for different shifts along the $y$-axis. The original uninformative prior (light blue) covers the entire plotted region.
• Figure 3: Prior dependence for the Gaussian example. The width of the likelihood is $2\times 10^{-4}$, which is between the narrowest and second narrowest priors considered. Left: Percent errors in the evidence computed with the LHME using importance resampled posterior draws, compared to the analytical solution as the strength of the prior is varied. Middle: Fractional ESS of the importance sampling. Right: Pareto-$\hat{k}$ diagnostic values for importance sampling. The evidence estimate is correctly flagged as unreliable when $\hat{k}$ is high (above 0.7).
• Figure 4: Same as Fig. \ref{['fig:gaussian_results']}, but for the Rosenbrock example with informative priors at different locations. Here ESS is low throughout and re-training $\varphi(\theta)$ is necessary. Pareto-$\hat{k} > 0.7$ successfully identifies the case where the evidence estimate is unreliable.