Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Poisson Observations to Fitted Negative Binomial Distribution

Yingying Yang, Niloufar Dousti Mousavi, Zhou Yu, Jie Yang

TL;DR

The paper addresses the challenge of distinguishing Poisson data from nearly Poisson-like negative binomial fits and the instability of NB MLE near Poisson conditions. It introduces an adaptive profile maximization algorithm (APMA) to robustly compute NB MLE under ν bounds and proposes a new extended NB parameterization with (μ, p) that includes Poisson as a limit, enabling consistent recovery of Poisson parameters from Poisson data. Theoretical results establish Poisson-consistent behavior under the extended NB framework and characterize MLE behavior under Poisson samples, while extensive numerical studies demonstrate APMA’s superior stability, speed, and accuracy compared with existing methods across diverse data regimes. Together, these contributions enhance practical Poisson–NB modeling by providing robust estimation procedures and a unifying parameterization that naturally covers Poisson, improving inference for count data.

Abstract

The negative binomial distribution has been widely used as a more flexible model than the Poisson distribution for count data. However, when the true data-generating process is Poisson, it is often challenging to distinguish it from a negative binomial distribution with extreme parameter values, and existing maximum likelihood estimation procedures for the negative binomial distribution may fail or produce unstable estimates. To address this issue, we develop a new algorithm for computing the maximum likelihood estimate of negative binomial parameters, which is more efficient and more accurate than existing methods. We further extend negative binomial distributions with a new parameterization to cover Poisson distributions as a special class. We provide theoretical justifications showing that, when applied to a Poisson data, the estimated parameters of the extended negative binomial distribution can consistently recover the true Poisson distribution.

From Poisson Observations to Fitted Negative Binomial Distribution

TL;DR

The paper addresses the challenge of distinguishing Poisson data from nearly Poisson-like negative binomial fits and the instability of NB MLE near Poisson conditions. It introduces an adaptive profile maximization algorithm (APMA) to robustly compute NB MLE under ν bounds and proposes a new extended NB parameterization with (μ, p) that includes Poisson as a limit, enabling consistent recovery of Poisson parameters from Poisson data. Theoretical results establish Poisson-consistent behavior under the extended NB framework and characterize MLE behavior under Poisson samples, while extensive numerical studies demonstrate APMA’s superior stability, speed, and accuracy compared with existing methods across diverse data regimes. Together, these contributions enhance practical Poisson–NB modeling by providing robust estimation procedures and a unifying parameterization that naturally covers Poisson, improving inference for count data.

Abstract

The negative binomial distribution has been widely used as a more flexible model than the Poisson distribution for count data. However, when the true data-generating process is Poisson, it is often challenging to distinguish it from a negative binomial distribution with extreme parameter values, and existing maximum likelihood estimation procedures for the negative binomial distribution may fail or produce unstable estimates. To address this issue, we develop a new algorithm for computing the maximum likelihood estimate of negative binomial parameters, which is more efficient and more accurate than existing methods. We further extend negative binomial distributions with a new parameterization to cover Poisson distributions as a special class. We provide theoretical justifications showing that, when applied to a Poisson data, the estimated parameters of the extended negative binomial distribution can consistently recover the true Poisson distribution.
Paper Structure (19 sections, 25 theorems, 72 equations, 3 figures, 15 tables, 2 algorithms)

This paper contains 19 sections, 25 theorems, 72 equations, 3 figures, 15 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Finding MLE for Negative Binomial Distribution
  3. Likelihood equation
  4. Efficient score function
  5. A new algorithm for negative binomial distributions
  6. Negative Binomial Estimates for Poisson Sample
  7. Other Parameterizations for NB Distributions
  8. Asymptotic results with other parameterizations in the literature
  9. A new parameterization and extended NB distributions
  10. Numerical Studies
  11. Comparison study on finding the MLE of NB distribution
  12. Further comparison between APMA and AZIAD
  13. Performance across three types of data regimes
  14. Conclusion
  15. Proofs
  16. ...and 4 more sections

Key Result

Theorem 2.1

Denote $M=\max\{Y_1, \ldots, Y_n\}$.

Figures (3)

  • Figure S.1: Histograms of the MLEs $\hat{\nu}$ (top) and $\hat{p}$ (bottom) across 1,000 Poisson$(10)$ random samples with various sample sizes.
  • Figure S.2: Histograms of the KS test statistic $D_n$ (top) and the critical number $d_n$ (bottom) across 1,000 Poisson($10$) random samples with various sample sizes.
  • Figure S.3: Scatter plots of $(\hat{\nu}, \hat{p})$ (top) along with its theoretical curve $p = \nu/(\nu+10)$, and scatter plots of $(D_n, d_n)$ (bottom) along with a reference line $d_n=D_n$ , based on 1,000 Poisson($10$) random samples.

Theorems & Definitions (29)

  • Example 2.1
  • Example 2.2
  • Theorem 2.1: simonsen1976solutionsimonsen1980correction
  • Theorem 2.2
  • Lemma 2.1
  • Example 2.3
  • Theorem 2.3
  • Example 2.4
  • Lemma 3.1
  • Theorem 3.1
  • ...and 19 more