Exact inference via quasi-conjugacy in two-parameter Poisson-Dirichlet hidden Markov models

TL;DR

This work addresses the challenge of inferring time-evolving, infinite-dimensional probability measures from unlabelled partition data by modeling the latent signal as a $ ext{PD}_{oldsymbol{eta},oldsymbol{ heta}}$ diffusion. By exploiting a Markov duality with a pure-death process on partitions and a coagulation algebra, the authors derive exact, finite-mixture forward–backward filters and smoothers, achieving fully tractable online and offline inference without MCMC or SMC. The framework yields closed-form posterior, predictive, and interpolated distributions, along with exact predictive functionals such as heterozygosity, and supports joint parameter estimation via likelihood-based methods. Demonstrations on synthetic data and a dynamic social network dataset illustrate substantial gains in accuracy and computation compared with bootstrap particle filtering, confirming practical scalability for unlabelled, partition-valued observations.

Abstract

We introduce a nonparametric model for time-evolving, unobserved probability distributions from discrete-time data consisting of unlabelled partitions. The latent process is a two-parameter Poisson-Dirichlet diffusion, and observations arise via exchangeable sampling. Applications include social and genetic data where only aggregate clustering summaries are observed. To address the intractable likelihood, we develop a tractable inferential framework that avoids label enumeration and direct simulation of the latent state. We exploit a duality between the diffusion and a pure-death process on partitions, together with coagulation operators that encode the effect of new data. These yield closed-form, recursive updates for forward and backward inference. We compute exact posterior distributions of the latent state at arbitrary times and predictive distributions of future or interpolated partitions. This enables online and offline inference and forecasting with full uncertainty quantification, bypassing MCMC and sequential Monte Carlo. Compared to particle filtering, our method achieves higher accuracy, lower variance, and substantial computational gains. We illustrate the methodology with synthetic experiments and a social network application, recovering interpretable patterns in time-varying heterozygosity.

Figures (13)

• Figure 1: Unlabelled partitions obtained over four intervals from the Infectious dataset.
• Figure 2: Left: Young diagrams of partitions in $\mathcal{P}_4$. Right: indexed coagulation of $\omega = (2,1,1)$ and $\gamma = (3,2,1,1)$, yielding $\mu=(4,3,2,1,1)$.
• Figure 3: Three examples of coagulation sets $\mathrm{coag}(\Omega, \Gamma)$ (orange) from $\Omega$ (yellow) and $\Gamma$ (red). Left: $\Omega = \{(1,1)\}$ and $\Gamma = \{(2)\}$ may produce $(2,1,1)$ or $(3,1)$. Right: $(2)$ in $\Gamma$ merges with $(1)$ or $(1^2)$, leading to $(3)$, $(2,1)$, $(2,1^{2})$, or $(3,1)$.
• Figure 4: Latent partition support in recursive filtering via prediction and update. Left: initial support $\Lambda = \{(2,1)\}$ at $t_0$. Center: prediction expands $\Lambda$ this to $L(\Lambda)$ using the dual process. Right: update with $\pi^1 = (1,1)$ produces $\mathrm{coag}(L(\Lambda), \pi^1)$ as the new support. Each node represents a component $\mathrm{PD}_{\alpha, \theta}^\lambda$.
• Figure 5: Smoothing via forward-backward latent structure. The forward latent partition support $\Lambda_{0:k}$ (left) is coagulated with the backward latent partition support $\Omega^{k:N}$ (right), to form the coagulation set $\mathrm{coag}(\Lambda_{0:k}, \Omega^{k:N})$ (middle). The latter supports the latent partitions in the posterior law of $X(t_k) | \Pi^{0:N}$.

Theorems & Definitions (17)

• Definition 1: Indexed coagulation
• Definition 2: General coagulation
• Proposition 1: Product expansion
• Proposition 2: Conditional predictive distribution
• Remark 1
• Proposition 3: Recursive filtering structure
• Theorem 1: Exact inference for PD-HMMs
• Definition 3: Coagulation of partitions
• Proposition 4
• Proposition 5