Learning from Neighbors with PHIBP: Predicting Infectious Disease Dynamics in Data-Sparse Environments

TL;DR

This work tackles predicting infectious disease counts in data-sparse settings by applying the Poisson Hierarchical Indian Buffet Process (PHIBP), a Bayesian framework that borrows information across counties through a global random measure $B_0$ and region-specific CRMs $B_j$. By exploiting a compound Poisson representation, the model yields exact posterior inference, enables robust prediction of unseen diseases, and supports Bayesian alpha-diversity $\mathscr{D}_j$ and beta-diversity $\mathscr{B}_{j,v}$ calculations based on localized posterior rates $\tilde{\sigma}_{j,l}(H_l)$. Empirical results on California data show that the GG subordinator substantially outperforms GA in log-likelihood and unseen-disease prediction, with geographic patterns in diversity measures (e.g., southern counties higher alpha-diversity; beta-diversity clustering near reference counties). The framework offers a principled, uncertainty-aware approach to real-time disease surveillance and is extended to multi-resolution, time-dynamic, and domain-adaptive extensions, highlighting the potential for broad applications beyond epidemiology.

Abstract

Modeling sparse count data, which arise across numerous scientific fields, presents significant statistical challenges. This chapter addresses these challenges in the context of infectious disease prediction, with a focus on predicting outbreaks in geographic regions that have historically reported zero cases. To this end, we present the detailed computational framework and experimental application of the Poisson Hierarchical Indian Buffet Process (PHIBP), with demonstrated success in handling sparse count data in microbiome and ecological studies. The PHIBP's architecture, grounded in the concept of absolute abundance, systematically borrows statistical strength from related regions and circumvents the known sensitivities of relative-rate methods to zero counts. Through a series of experiments on infectious disease data, we show that this principled approach provides a robust foundation for generating coherent predictive distributions and for the effective use of comparative measures such as alpha and beta diversity. The chapter's emphasis on algorithmic implementation and experimental results confirms that this unified framework delivers both accurate outbreak predictions and meaningful epidemiological insights in data-sparse settings.

Figures (5)

• Figure 1: (Left) test log-likelihoods of GG and GA PHIBP models. (Middle) posterior distribution of $\alpha_0$. (Right) posterior distribution of $\alpha_{37}$ (corresponding to San Francisco).
• Figure 2: Predicted vs test data counts for county-disease pairs with 0 counts in training dataset.
• Figure 3: Heatmap of posterior mean of pairwise beta-diversities for GG with reference counties (in dark blue) as Del Norte (left) and San Diego (right).
• Figure 4: Heatmap of posterior mean of alpha-diversities for GG (left) and GA (right).
• Figure 5: Heatmap of posterior precision of alpha-diversities for GG (left) and average county population from 2001-2014 (right); note the logarithmic scale.