ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model

TL;DR

ItDPDM introduces a discrete Poisson diffusion model that models non-negative discrete data natively and enables exact likelihood estimation through the Poisson Reconstruction Loss (PRL). By leveraging the information-theoretic Poisson diffusion framework and the I-MPRL identity, the method provides a principled, non-variational route to training and likelihood evaluation, with thermodynamic integration linking PRL to the true data likelihood. Empirical results on synthetic distributions and real data (CIFAR-10 and Lakh MIDI) show ItDPDM achieves lower NLL and competitive generation quality, demonstrating its potential for distribution-robust discrete generative modeling. While a proof-of-concept, the approach lays groundwork for exact-likelihood discrete diffusion with scalable continuous-time sampling and highlights avenues for future improvements in training scale and architectural design.

Abstract

Generative modeling of non-negative, discrete data, such as symbolic music, remains challenging due to two persistent limitations in existing methods. Firstly, many approaches rely on modeling continuous embeddings, which is suboptimal for inherently discrete data distributions. Secondly, most models optimize variational bounds rather than exact data likelihood, resulting in inaccurate likelihood estimates and degraded sampling quality. While recent diffusion-based models have addressed these issues separately, we tackle them jointly. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), inspired by photon arrival process, which combines exact likelihood estimation with fully discrete-state modeling. Central to our approach is an information-theoretic Poisson Reconstruction Loss (PRL) that has a provable exact relationship with the true data likelihood. ItDPDM achieves improved likelihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, ItDPDM attains superior likelihood estimates and competitive generation quality-demonstrating a proof of concept for distribution-robust discrete generative modeling.

Figures (19)

• Figure 1: Classification of diffusion models based on latent state-space (DS/CS) and timesteps (DT/CT), resulting in 4 combinations - DTCS, CTCS, DTDS, and CTDS
• Figure 2: Unconditional image samples generated by ItDPDM
• Figure 3: Gaussian diffusion fails to accurately learn the discrete probability density
• Figure 4: Comparison of Gaussian (top) and Poisson diffusion processes (bottom).
• Figure 5: Poisson Reconstruction Loss (PRL): (a) vs. denoised pixel $\hat{x}$, for fixed ground truth pixel 1; (b) vs. ground truth pixel $x$, for fixed denoised output 1.

Theorems & Definitions (15)

• Lemma 1: Poisson Reconstruction Loss
• Lemma 2: Linearity in Poisson Channel
• Lemma 3
• Lemma 4: Optimal Estimator in Poisson Channel
• Lemma 5
• Lemma 6
• Lemma 7
• Lemma 8
• Lemma 9
• Lemma 10