Locally sparse varying coefficient mixed model with application to longitudinal microbiome differential abundance

TL;DR

This paper tackles longitudinal differential abundance analysis in microbiome studies by introducing LSVCMM, a locally sparse varying coefficient mixed model that jointly models time-varying effects and within-subject correlation. It estimates ${\boldsymbol \beta}(\cdot)$ via penalized kernel smoothing and enforces both local and global sparsity through an adaptive sparse group Lasso penalty ${\mathcal P}_{\lambda,\alpha}({\mathbf B}; \Omega)$, while accommodating irregular sampling with a parametric working covariance for the random effects. The authors demonstrate via simulations that accounting for temporal dependence improves estimation and model selection, and they apply LSVCMM to a mouse oral cancer study to uncover weekspecific differential abundances and genotype-diagnosis interactions that cross-sectional analyses miss. The work provides an R implementation, EBIC-based tuning, and bootstrap simultaneous confidence bands, offering a practical tool for time-resolved microbiome differential analysis with irregular designs and missing data, along with a discussion of limitations and directions for future extension to zero-inflated counts and multivariate dependencies.

Abstract

Differential abundance (DA) analysis in microbiome studies has recently been used to uncover a plethora of associations between microbial composition and various health conditions. While current approaches to DA typically apply only to cross-sectional data, many studies feature a longitudinal design to better understand the underlying microbial dynamics. To study DA in longitudinal microbial studies, we introduce a novel varying coefficient mixed-effects model with local sparsity. The proposed method can identify time intervals of significant group differences while accounting for temporal dependence. Specifically, we exploit a penalized kernel smoothing approach for parameter estimation and include a random effect to account for serial correlation. In particular, our method operates effectively regardless of whether sampling times are shared across subjects, accommodating irregular sampling and missing observations. Simulation studies demonstrate the necessity of modeling dependence for precise estimation and support recovery. The application of our method to a longitudinal study of mice oral microbiome during cancer development revealed significant scientific insights that were otherwise not discernible through cross-sectional analyses. An R implementation is available at https://github.com/fontaine618/LSVCMM.

Figures (4)

• Figure 1: Evaluation metrics in the missing data scenario reported as the mean (line) and standard error (band) across 100 replications.
• Figure 2: Evaluation metrics in the irregular sampling scenario reported as the mean (line) and standard error (band) across 100 replications.
• Figure 3: (Left) Number of observed and missing samples per week, stratified by genotype and diagnosis. (Right) Patterns of missingness; row labels indicate the frequency of that pattern occurring among the 65 mice.
• Figure 4: Estimated main effects (top: KO less WT; middle: SCC less ED/CIS) and interaction (bottom: coded as 1 if KO and SCC, and 0 otherwise). White cells correspond to a zero estimate, colored cells correspond to a non-zero estimate. Asterisks indicate a time point where the 95% simultaneous confidence band excludes zero. Columns represent estimate emerging from three different methods. Only OTUs with a significant difference for at least one time point and one method are included in each row (out of 187 total OTUs).