Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate posterior recalibration

Tiffany Cai, Philip Greengard, Ben Goodrich, Andrew Gelman

Abstract

Bayesian inference is often implemented using approximations, which can yield interval estimates that are too narrow, not fully capturing the uncertainty in the posterior distribution. We address the question of how to adjust these approximate posteriors so that they appropriately capture uncertainty. vWe introduce two methods that extend simulation-based calibration checking (SBC) to widen approximate posterior uncertainty intervals to aim for marginal calibration. We demonstrate these methods in several experimental settings, and we discuss the challenge of calibration using posterior inferences and the potential for posterior recalibration of hierarchical models.

Approximate posterior recalibration

Abstract

Bayesian inference is often implemented using approximations, which can yield interval estimates that are too narrow, not fully capturing the uncertainty in the posterior distribution. We address the question of how to adjust these approximate posteriors so that they appropriately capture uncertainty. vWe introduce two methods that extend simulation-based calibration checking (SBC) to widen approximate posterior uncertainty intervals to aim for marginal calibration. We demonstrate these methods in several experimental settings, and we discuss the challenge of calibration using posterior inferences and the potential for posterior recalibration of hierarchical models.
Paper Structure (20 sections, 19 equations, 6 figures, 4 tables, 2 algorithms)

This paper contains 20 sections, 19 equations, 6 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Simulation-based calibration checking
  3. Proposed procedures for approximate posterior calibration (APC)
  4. Nominal coverage method
  5. Method using $z$-scores
  6. Comparison
  7. Calibration experiments
  8. A simple example: a one-parameter Gaussian model
  9. Hierarchical model (8 schools) and ADVI
  10. Posterior recalibration
  11. Understanding posterior recalibration for a one-parameter normal model
  12. Prior calibration checking
  13. Posterior calibration checking
  14. Recalibration using a location-scale shift
  15. Evaluating posterior recalibration
  16. ...and 5 more sections

Figures (6)

  • Figure 1: SBC quantiles for model \ref{['simple_model']} when sampling from the exact posterior (left) and from posterior with sampling error (right)
  • Figure 2: SBC quantiles for the 8 schools model (Section \ref{['sec:8schools']}) when sampling with HMC (left) and ADVI (right). The HMC posterior is calibrated, while the ADVI posterior is not.
  • Figure 3: Posterior mean and standard deviation for ADVI vs. HMC. Each dot represents a different replication from the prior predictive distribution (i.e. a different $l\in\{1,\ldots,L\}$ in Algorithm \ref{['alg:adjust']}).
  • Figure 4: Distribution of $z$-scores and quantiles in simulation-based calibration checking when the correct model is being fit for the simple normal example, under two scenarios, both for $\sigma=1$. Top row: Simulating the parameter $\theta^l$ from the prior distribution. Bottom row: Simulating $\theta^l$ from the posterior distribution, in which case the distributions depend on the data, $y$. Curves show distributions conditional on three possible data values.
  • Figure 5: Simulation-based calibration checking using 200 replications of 1000 simulation draws, for each row, with $\sigma=1$. Top row: Simulating the parameter $\theta^l$ from the prior distribution. Center and bottom rows: Simulating $\theta^l$ from the posterior distribution, in which case the distributions depend on the data, $y$. As predicted by theory, the prior predictive simulations show calibration and the posterior predictive simulations do not, for $\sigma=1$.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Remark 3.1: Adjusted posterior satisfies self-consistency property in \ref{['eq:z_score_condition']}
  • Remark 5.1: Derivation of \ref{['eq:from_posterior']}