Calibrating hierarchical Bayesian domain inference for a proportion

TL;DR

The paper addresses calibrated Bayesian domain inference for proportions in SAE by extending FAB intervals from normally distributed outcomes to binary data and integrating with MRP to correct sample bias. It develops a two-step computation to construct FAB Wald, AC, and Wilson intervals for proportions, including an all-in penalty to improve behavior near the boundaries, and evaluates their frequentist coverage via simulation and a COVID-19 infection-rate application. Across simulations and the real data example, FAB intervals achieve improved domain-specific coverage relative to standard Bayesian credible or classical intervals, though at the cost of wider interval lengths. The work provides practical, calibrated interval tools for region- and subgroup-specific proportion estimates in SAE, with implications for targeted policy and resource allocation in public health and related fields.

Abstract

Small area estimation (SAE) improves estimates for local communities or groups, such as counties, neighborhoods, or demographic subgroups, when data are insufficient for each area. This is important for targeting local resources and policies, especially when national-level or large-area data mask variation at a more granular level. Researchers often fit hierarchical Bayesian models to stabilize SAE when data are sparse. Ideally, Bayesian procedures also exhibit good frequentist properties, as demonstrated by calibrated Bayes metrics. However, hierarchical Bayesian models tend to shrink domain estimates toward the overall mean and may produce credible intervals that do not maintain nominal coverage. Hoff et al. developed the Frequentist, but Assisted by Bayes (FAB) intervals for subgroup estimates with normally distributed outcomes. However, non-normally distributed data present new challenges, and multiple types of intervals have been proposed for estimating proportions. We examine domain inference with binary outcomes and extend FAB intervals to improve nominal coverage. We describe how to numerically compute FAB intervals for a proportion and evaluate their performance through repeated simulation studies. Leveraging multilevel regression and poststratification (MRP), we further refine SAE to correct for sample selection bias, construct the FAB intervals for MRP estimates and assess their repeated sampling properties. Finally, we apply the proposed inference methods to estimate COVID-19 infection rates across geographic and demographic subgroups. We find that the FAB intervals improve nominal coverage, at the cost of wider intervals.

Figures (7)

• Figure 1: The coverage probability of Bayesian 95% credible intervals of $\theta_i$ as a function of $\theta_i$ integrating out all responses under a hierarchical Bayes model: $y_i \sim \textrm{Binomial}(\cdot \mid \theta_i, n_i);\,\,\, \theta_i = \textrm{inv\_logit}(\eta_i);\,\,\, \eta_i \sim \textrm{N}(\cdot \mid 0, 1)$, with different $n_i$ values. The dashed horizontal line indicates the value of 0.95.
• Figure 2: The coverage probability for the various intervals under fixed $n_i = 100$ and varying $\theta_i$. A solid black line indicates the Frequentist assisted Bayes (FAB) interval, a dashed line indicates the FAB interval version with all-in penalty, and a gray dot dash line indicates the originally defined interval.
• Figure 3: The range of $y_i$'s values for which an Frequentist assisted Bayes (FAB) or confidence interval built on $y_i / n_i$ captures a given $\theta_i$. The FAB intervals created from $y_i$'s within the black solid lines, the FAB (all-in penalty) intervals created from $y_i$'s within the black dashed lines, and the original intervals created from $y_i$'s within the gray dotted lines include $\theta_i$. The histograms in the plots display $p(y_i \mid \theta_i)$, i.e., the associated probabilities of $y_i$, whose any of three intervals under comparison cover $\theta_i$.
• Figure 4: Coverage probability of the risk interval $\mathcal{I}^{F}\left(\theta, s_i, \textrm{var}(\theta)\right)$ in gray region and the determination interval $\mathcal{I}^{F}\left(\theta, s_i, \sqrt{\frac{\theta(1 - \theta)}{n_i}}\right)$ in dark gray for $\theta_i \in [0, 1]$. for the Wilson Frequentist assisted Bayes (FAB) interval and Wilson FAB interval with all-in penalty. Both intervals are plotted based on the optimum $s_i$. The black horizontal line is $\widehat{\theta}_i$ = 0.1, i.e., the value we want to build an interval around, and the bounds of the thicker black line represent the FAB intervals for this $\widehat{\theta}_i$.
• Figure 5: The coverage probability and average interval lengths of Bayesian credible intervals (CIs) and Frequentist assisted Bayes (FAB) intervals of the ZIP-level estimates with different prior specifications. The left panel is based on data simulated under the independent prior, and the right panel is for data simulated under the spatial prior. For the Wilson, Wald, and Agresti-Coull (AC) interval type, the FAB intervals based on the spatial prior are displayed in black on the left whereas the FAB intervals based on the global-local normal prior are colored in light gray on the right. The credible intervals are colored in semi-dark gray and on the left of the plot, with the left one under the spatial prior and the right one under the global-local prior. The dashed line represents 95% coverage probability.