Understanding When Poisson Log-Normal Models Outperform Penalized Poisson Regression for Microbiome Count Data

Abstract

Multivariate count models are often justified by their ability to capture latent dependence, but researchers receive little guidance on when this added structure improves on simpler penalized marginal Poisson regression. We study this question using real microbiome data under a unified held-out evaluation framework. For count prediction, we compare PLN and GLMNet(Poisson) on 20 datasets spanning 32 to 18,270 samples and 24 to 257 taxa, using held-out Poisson deviance under leave-one-taxon-out prediction with 3-fold sample cross-validation rather than synthetic or in-sample criteria. For network inference, we compare PLNNetwork and GLMNet(Poisson) neighborhood selection on five publicly available datasets with experimentally validated microbial interaction truth. PLN outperforms GLMNet(Poisson) on most count-prediction datasets, with gains up to 38 percent. The primary predictor of the winner is the sample-to-taxon ratio, with mean absolute correlation as the strongest secondary signal and overdispersion as an additional predictor. PLNNetwork performs best on broad undirected interaction benchmarks, whereas GLMNet(Poisson) is better aligned with local or directional effects. Taken together, these results provide guidance for choosing between latent multivariate count models and penalized Poisson regression in biological count prediction and interaction recovery.

Figures (6)

• Figure 1: Conceptual contrast between earlier evaluation practice and the count-prediction protocol used in this study. Previous work often assessed count models with in-sample fit criteria computed on the observed abundance matrix. Our benchmark instead withholds a target taxon, predicts its counts out of sample from the remaining taxa, and scores those predictions with held-out Poisson deviance under leave-one-taxon-out cross-validation.
• Figure 2: Dataset-level winner as a function of $N/D$ and MAC across the 20-dataset real-data benchmark. Blue circles: PLN achieves lower held-out Poisson deviance. Red squares: GLMNet(Poisson) achieves lower held-out Poisson deviance. The orange triangle marks American Gut 1, a GLMNet win at low $N/D$ and high MAC that runs counter to the general pattern.
• Figure 3: Representative datasets illustrating both directions of the PLN vs. GLMNet(Poisson) comparison. Top: datasets where PLN achieves lower held-out Poisson deviance. Bottom: datasets where GLMNet(Poisson) achieves lower held-out Poisson deviance. Each panel reports held-out Poisson deviance (mean $\pm$ SE across three outer cross-validation folds) for PLN, GLMNet(Poisson), and the featureless baseline. The annotated difference and $p$-value are from the paired comparison of PLN and GLMNet(Poisson) across folds.
• Figure 4: Edge-recovery F1 score on five experimental ground-truth datasets for PLNNetwork and GLMNet(Poisson). Datasets are grouped by ground-truth type: broad edge recovery (OMM12, OMM12 keystone 2023, PairInteraX) and local/directional (butyrate assembly 2021, host fitness 2018). Bars show mean F1 score over $B = 20$ bootstrap resamples of the abundance matrix; error bars are $\pm 1$ SE.
• Figure 5: Ground-truth and inferred networks for the OMM12 defined community (12 taxa). Each panel shows the experimental ground truth (left), the PLNNetwork prediction (centre), and the GLMNet(Poisson) prediction (right). Node labels are strain codes: KB1 (E. faecalis), YL2 (B. animalis), KB18 (A. muris), YL27 (M. intestinale), YL31 (F. plautii), YL32 (E. clostridioformis), YL44 (A. muciniphila), YL45 (T. muris), I46 (C. innocuum), I48 (B. caecimuris), I49 (L. reuteri), YL58 (B. coccoides). Edge colour indicates interaction sign (blue: positive, orange: negative).

Theorems & Definitions (2)

• Proposition 1: Properness of Poisson deviance
• proof