Statistical Guarantees for Data-driven Posterior Tempering

TL;DR

This work analyzes data-driven tempering in Bayesian inference by studying α-posterior posteriors where the likelihood is raised to a data-dependent power $\hat{α}_n$. It proves that, in the regime $\frac{1}{n} \ll \hat{α}_n \ll 1$, the posterior moments converge to those of a normal distribution and establishes a Bernstein–von Mises-type result; it further shows that the posterior mean is asymptotically normal when $\frac{1}{\sqrt{n}} \ll \hat{α}_n \ll 1$ and provides a novel Laplace approximation to quantify the distance between the posterior mean and the MLE. The paper also treats mixed regimes where $\hat{α}_n$ has a mass at infinity with positive probability, yielding a BvM-type theorem for a mixed $\hat{α}_n^{\text{mix}}$-posterior. These results extend existing theory for fixed α-posterior to realistic, data-driven tempering schemes arising in cross-validation and calibration procedures, with implications for robust Bayesian inference under model misspecification and for understanding the asymptotics of modern tempering-based methods.

Abstract

Posterior tempering reduces the influence of the likelihood in the calculation of the posterior by raising the likelihood to a fractional power $α$. The resulting power posterior - also known as an $α$-posterior or fractional posterior - has been shown to exhibit appealing properties, including robustness to model misspecification and asymptotic normality (Bernstein-von Mises theorem). However, practical recommendations for selecting the tempering parameter and statistical guarantees for the resulting power posterior remain open questions. Cross-validation-based approaches to tuning this parameter suggest interesting asymptotic regimes for the selected $α$, which can either vanish or behave like a mixture distribution with a point mass at infinity and the remaining mass converging to zero. We formalize the asymptotic properties of the power posterior in these regimes. In particular, we provide sufficient conditions for (i) consistency of the power posterior moments and (ii) asymptotic normality of the power posterior mean. Our analysis required us to establish a new Laplace approximation that is interesting in its own right and is the key technical tool for showing a critical threshold $α\asymp 1/\sqrt{n}$ where the asymptotic normality of the posterior mean breaks. Our results allow for the power to depend on the data in an arbitrary way.

Figures (9)

• Figure 3: Proportion of corner solutions vs. sample size ($n$).
• Figure 4: The squared norm of $\sqrt{n}(\hat{\beta}^\text{B} - \hat{\beta})$ (left) and $\sqrt{n}(\hat{\beta}^\text{B} - \beta^*)$ (right) averaged over 100 replications, as a function of sample size. The $\alpha_n$-posterior mean was obtained by samples from the $\alpha_n$-posterior, where $\alpha_n = 0.5n^{-3/4}$ (gray), $\alpha_n = 0.5n^{-1/2}$ (orange), and $\alpha_n = 0.5n^{-1/4}$ (blue). The data was generated from the i.i.d. model $\mathbb{P}(Y_i = 1|x_i) = [1 + \exp(-x_i^\top \beta^*)]^{-1}$, where $\{x_i\} \overset{iid}{\sim}\mathcal{N}(0, I_3)$ and $\beta^* =[1, -0.5, 0.1]$.
• Figure 5: Panel A and B: dependence of $\hat{\alpha}_n$ (y-axis) on $n$ (x-axis) in the well-specified model setting (A) and misspecified model setting (B). Points correspond to $\hat{\alpha}_n$ values averaged over 1,000 replications with overlaid curves of best fit. Plot titles denote the estimated stochastic order of $\hat{\alpha}_n$. Corner solutions at $\hat{\alpha}_n=\infty$ are discarded in the plotting and estimation in the following settings: BCV/well-specified, BCV+VI/well-specified, train-test/well-specified and misspecified.
• Figure 6: Panel A, B, and C: dependence of $\hat{\alpha}_n$ (y-axis) on $n$ (x-axis) under the full model setting (A), model excluding ethnicity (B), and model excluding squared experience. Points correspond to $\hat{\alpha}_n$ values (with corner solutions discarded) averaged over $100$ replications with overlaid curves of best fit. Plot titles denote the estimated stochastic order of $\hat{\alpha}_n$. The plot for BCV + VI under the model excluding squared experience is not plotted because "non-corner solutions" only appear for $n = 100$. We also not that the "non-corner" solutions according to LOOCV appear to be growing -- rather than shrinking -- with $n$. We include this result for completeness.
• Figure 7: Figure \ref{['fig:lim_dist_mix']} with the $\hat{\alpha}_n$ values plotted on a linear scale. Different x-axis scales are used for $n=100$, $n=1,000$, and $n=5,000$ to showcase the different scales of the selected $\hat{\alpha}_n$ values as $n$ grows.

Theorems & Definitions (46)

• Theorem 1
• Corollary 1
• Proposition 1
• Theorem 2
• Corollary 2
• Remark 1
• Lemma 1
• Proposition 2
• Proposition 3
• Theorem 3