Improving the Accuracy of Amortized Model Comparison with Self-Consistency

TL;DR

The paper tackles the instability of amortized Bayesian model comparison under model misspecification by evaluating self-consistency (SC) training across four amortized approaches. It shows that estimators based on approximating parameter posteriors (NPE, NPLE) generally yield more accurate marginal likelihoods and Bayes factors than direct evidence or PMP methods (NEE, NPMP), with SC offering the most robust gains when test data remain near SC-trained distributions. Across synthetic and real-world case studies, NPE with SC training closely tracks gold-standard baselines like bridge sampling, especially under moderate misspecification, while NPLE’s benefits are more conditional and scenario-dependent. The results provide practical guidance to prefer parameter-posterior-based amortized methods and augment them with SC training on empirical data to mitigate extrapolation bias in misspecified settings, while highlighting limits in likelihood-free regimes and suggesting avenues for future enhancements through iterative training strategies.

Abstract

Amortized Bayesian inference (ABI) offers fast, scalable approximations to posterior densities by training neural surrogates on data simulated from the statistical model. However, ABI methods are highly sensitive to model misspecification: when observed data fall outside the training distribution (generative scope of the statistical models), neural surrogates can behave unpredictably. This makes it a challenge in a model comparison setting, where multiple statistical models are considered, of which at least some are misspecified. Recent work on self-consistency (SC) provides a promising remedy to this issue, accessible even for empirical data (without ground-truth labels). In this work, we investigate how SC can improve amortized model comparison conceptualized in four different ways. Across two synthetic and two real-world case studies, we find that approaches for model comparison that estimate marginal likelihoods through approximate parameter posteriors consistently outperform methods that directly approximate model evidence or posterior model probabilities. SC training improves robustness when the likelihood is available, even under severe model misspecification. The benefits of SC for methods without access of analytic likelihoods are more limited and inconsistent. Our results suggest practical guidance for reliable amortized Bayesian model comparison: prefer parameter posterior-based methods and augment them with SC training on empirical datasets to mitigate extrapolation bias under model misspecification.

Figures (9)

• Figure 1: Case Study 1: Estimating Marginal Likelihood of Multivariate Gaussian. Analytic ($x$-axis) vs. Approximate ($y$-axis), log marginal likelihood of the multivariate gaussian. NPE+SC achieves high accuracy, while NPLE and NEE only do so if combined with training on data generated from a wider prior (helping the likelihood/evidence networks to be more robust against prior misspecification).
• Figure 2: Case Study 2: Estimating Bayes Factors of Multivariate Gaussian. Analytic ($x$-axis) vs. Approximate ($y$-axis), log Bayes factor of the multivariate gaussian. In general, methods approximating the marginal likelihoods indirectly through parameter posteriors (NPE, NPLE) outperform direct approaches (NEE, NPMP). NPE+SC achieves high accuracy when the data is similar to that used during training; outside of that, SC training helps but does not fully close the gap. NPLE+SC shows similar pattern, but with a slightly worse performance. NEE and NPMP are poorly calibrated outside of the simulation-based training data, regardless of SC training.
• Figure 3: Case Study 3: Racing Diffusion Model of Decision Making. Log marginal likelihood estimated with bridge sampling ($x$-axis) vs neural estimates ($y$-axis) for the 17 participants in the lexical decision task wagenmakers2008diffusion. Columns show two models estimated with NPE or NPLE. Top row shows results without SC training, middle row SC training on a data subset (10 participants), and bottom row SC training on all 17 participants. NPE and NPLE without SC training (top row) underestimate the log marginal likelihood. When data from all participants are used for SC training (bottom row), NPE+SC gives essentially correct results. When some participants are left out from SC training (middle row), the error of NPE+SC is slightly larger for these participants. NPLE does not appear to benefit from SC training. These results are consistent for both statistical models $M_0$ and $M_1$.
• Figure 4: Case Study 4: Air Passenger Traffic Forecasting. Log marginal likelihood estimated with bridge sampling ($x$-axis) vs neural estimates ($y$-axis) for the 15 countries of the air passenger traffic dataset. Each column shows results for one of the four statistical models fit on the data. Top row shows results for NPE without SC training, middle row NPE with SC training on a data subset (8 countries), and bottom row NPE with SC training on all 15 countries). NPE without SC training (top row) underestimates the log marginal likelihood. When data from all countries are used for SC training (bottom row), NPE+SC gives essentially correct results. When some of the countries are left out from SC training (middle row), the error of NPE+SC is slightly larger for these countries. These results are consistent for all four statistical models (columns in the figure).
• Figure 5: Case Study 1: Estimating Marginal Likelihood of Multivariate Gaussian. Relative error of the approximate log marginal likelihood of the Multivariate Gaussian case study. The top panel shows results tested on data falling in the generative scope of the model. Middle panel shows results where the model is misspecified but the data corresponds to the one used for SC training. The bottom panel shows results where the test data is even further than that during SC training. The box-plots summarize the distribution over 256 simulations in each cell.