Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression

Mingxuan Wang, Cheng Chen, Gaoyang Jiang, Zijia Ren, Chuangxin Zhao, Lu Shi, Yanbiao Ma

TL;DR

ScDiVa introduces masked discrete diffusion for single-cell data, aligning generative modeling with dropout and eliminating autoregressive ordering biases. By combining entropy-normalized serialization, a latent anchor, depth-invariant sampling, and a dual denoising objective, it jointly models gene identity and expression magnitude within a bidirectional Transformer. Pre-trained on tens of millions of cells, ScDiVa achieves strong transfer across batch integration, cell type annotation, perturbation prediction, and GRN inference, outperforming state-of-the-art baselines. The work demonstrates that discrete diffusion with an absorbing state is a biologically principled alternative to autoregressive approaches for high-dimensional, sparse, and unordered single-cell profiles, with potential impact on scalable cellular representation learning and downstream biology.

Abstract

Single-cell RNA-seq profiles are high-dimensional, sparse, and unordered, causing autoregressive generation to impose an artificial ordering bias and suffer from error accumulation. To address this, we propose scDiVa, a masked discrete diffusion foundation model that aligns generation with the dropout-like corruption process by defining a continuous-time forward masking mechanism in token space. ScDiVa features a bidirectional denoiser that jointly models discrete gene identities and continuous values, utilizing entropy-normalized serialization and a latent anchor token to maximize information efficiency and preserve global cell identity. The model is trained via depth-invariant time sampling and a dual denoising objective to simulate varying sparsity levels while ensuring precise recovery of both identity and magnitude. Pre-trained on 59 million cells, scDiVa achieves strong transfer performance across major benchmarks, including batch integration, cell type annotation, and perturbation response prediction. These results suggest that masked discrete diffusion serves as a biologically coherent and effective alternative to autoregression.

ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression

TL;DR

ScDiVa introduces masked discrete diffusion for single-cell data, aligning generative modeling with dropout and eliminating autoregressive ordering biases. By combining entropy-normalized serialization, a latent anchor, depth-invariant sampling, and a dual denoising objective, it jointly models gene identity and expression magnitude within a bidirectional Transformer. Pre-trained on tens of millions of cells, ScDiVa achieves strong transfer across batch integration, cell type annotation, perturbation prediction, and GRN inference, outperforming state-of-the-art baselines. The work demonstrates that discrete diffusion with an absorbing state is a biologically principled alternative to autoregressive approaches for high-dimensional, sparse, and unordered single-cell profiles, with potential impact on scalable cellular representation learning and downstream biology.

Abstract

Single-cell RNA-seq profiles are high-dimensional, sparse, and unordered, causing autoregressive generation to impose an artificial ordering bias and suffer from error accumulation. To address this, we propose scDiVa, a masked discrete diffusion foundation model that aligns generation with the dropout-like corruption process by defining a continuous-time forward masking mechanism in token space. ScDiVa features a bidirectional denoiser that jointly models discrete gene identities and continuous values, utilizing entropy-normalized serialization and a latent anchor token to maximize information efficiency and preserve global cell identity. The model is trained via depth-invariant time sampling and a dual denoising objective to simulate varying sparsity levels while ensuring precise recovery of both identity and magnitude. Pre-trained on 59 million cells, scDiVa achieves strong transfer performance across major benchmarks, including batch integration, cell type annotation, and perturbation response prediction. These results suggest that masked discrete diffusion serves as a biologically coherent and effective alternative to autoregression.
Paper Structure (67 sections, 1 theorem, 34 equations, 11 figures, 5 tables, 2 algorithms)

This paper contains 67 sections, 1 theorem, 34 equations, 11 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-Based Single-Cell Pre-training
  4. Continuous Probabilistic Representation Learning
  5. Diffusion Models and Masked Generation
  6. ScDiVa: Diffusion Foundation Model for Single Cells
  7. Problem Formulation and Diffusion Modeling
  8. Cell Representation and Unified Embedding
  9. Depth-Invariant Sampling and Optimization
  10. Model Architecture and Pre-training Protocol
  11. Experiments
  12. Downstream Tasks and Comparative Methods
  13. Rank-Value Joint Reconstruction
  14. Multi-batch Integration
  15. Cell Type Annotation
  16. ...and 52 more sections

Key Result

Theorem 1.1

Define an observation map $\phi:\mathcal{V}\cup{\varnothing}\rightarrow \mathcal{V}\cup{0}$ by $\phi(g)=g$ for $g\in\mathcal{V}$ and $\phi(\varnothing)=0$ (interpreting [MASK] as complete information loss). For each position $i$, define the observed "signal" random variable $y_t^i=\phi(x_t^i)$. Then In particular, as $t\to 1$, $q(y_t^i\mid x_0^i)\Rightarrow \delta(y_t^i,0)$, i.e., complete dropout

Figures (11)

  • Figure 1: Overview of the scDiVa Architecture. The framework employs a masked modeling approach with a Latent Encoder to capture global cell contexts. The input gene expression profile is randomly masked and processed through a 12-layer Transformer encoder equipped with RoPE attention and SwiGLU activation. The model optimizes a dual objective ($\mathcal{L}$), combining Cross-Entropy ($\mathcal{L}_{CE}$) for gene identity reconstruction and Mean Squared Error ($\mathcal{L}_{MSE}$) for expression value regression.
  • Figure 2: Visual comparison of generative dynamics. From left to right: Zero-Mask Baseline (Ground Truth), Single-Step Inference, and 32-Step Diffusion Reconstruction.
  • Figure 3: Visualization of multi-batch integration on the Immune dataset. The UMAP projections illustrate the latent representations learned by scDiVa. The left panel is colored by batch source, demonstrating the effective removal of batch effects (mixing). The right panel is colored by cell type, highlighting the distinct preservation of biological clusters and cellular identities.
  • Figure 4: Comprehensive evaluation of scDiVa performance. (a) Cross-batch full fine-tuning performance (Accuracy and Macro-F1) on hPancreas, MS, Myeloid, and Myeloid_b datasets. (b) Zero-shot cell annotation performance (Accuracy and F1 Score) achieved by freezing the backbone and training only the MLP head across various datasets. (c) Perturbation prediction comparison against other models on the Adamson (left) and Norman (right) datasets, evaluated using DE $\Delta$ Correlation and DE MSE metrics.
  • Figure 5: Predicted vs. observed expression shifts in the Adamson dataset. Distributional changes for the top 20 differentially expressed genes under DAD1 and AMIGO3 perturbations.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Theorem 1.1: Dropout--Diffusion Isomorphism
  • proof