PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer

TL;DR

The paper presents PanFoMa, a lightweight hybrid foundation model for pan-cancer single-cell transcriptomics that combines a local Transformer encoder with a global Mamba decoder and a dynamic gene-sorting mechanism. It also introduces PanFoMaBench, a large-scale benchmark with over 3.5 million cells across 33 cancer subtypes to evaluate cross-dataset generalization. Empirically, PanFoMa outperforms multiple baselines across pan-cancer diagnosis, gene regulatory network inference, batch integration, cell-type annotation, and multi-omic integration, demonstrating the value of the local-to-global design. This work provides a scalable, practical foundation model and benchmark to accelerate precision oncology and integrative single-cell analyses.

Abstract

Single-cell RNA sequencing (scRNA-seq) is essential for decoding tumor heterogeneity. However, pan-cancer research still faces two key challenges: learning discriminative and efficient single-cell representations, and establishing a comprehensive evaluation benchmark. In this paper, we introduce PanFoMa, a lightweight hybrid neural network that combines the strengths of Transformers and state-space models to achieve a balance between performance and efficiency. PanFoMa consists of a front-end local-context encoder with shared self-attention layers to capture complex, order-independent gene interactions; and a back-end global sequential feature decoder that efficiently integrates global context using a linear-time state-space model. This modular design preserves the expressive power of Transformers while leveraging the scalability of Mamba to enable transcriptome modeling, effectively capturing both local and global regulatory signals. To enable robust evaluation, we also construct a large-scale pan-cancer single-cell benchmark, PanFoMaBench, containing over 3.5 million high-quality cells across 33 cancer subtypes, curated through a rigorous preprocessing pipeline. Experimental results show that PanFoMa outperforms state-of-the-art models on our pan-cancer benchmark (+4.0\%) and across multiple public tasks, including cell type annotation (+7.4\%), batch integration (+4.0\%) and multi-omics integration (+3.1\%). The code is available at https://github.com/Xiaoshui-Huang/PanFoMa.

Figures (4)

• Figure 1: Comparison of different architectures for modeling single-cell gene expression. (a) Transformer-based methods capture local features but incur $O(n^2)$ complexity. (b) Mamba-based models offer $O(n)$ efficiency but require fixed input ordering. (c) Our proposed PanFoMa captures both local and global dependencies via a hybrid encoder-decoder design, resulting in an overall computational complexity of $O(C \cdot M^2 + N \log N)$, where $N=C \cdot M$.
• Figure 2: Overview of the PanFoMa algorithm. Firstly, the Local-context encoder first divide the input into chunks and encodes local gene interactions within transcriptome chunks using shared Transformer layers. Second, the global sequential feature decoder guides dynamic reordering of all genes based on relevance. The reordered sequence is processed by a bidirectional Mamba decoder, and a gating module fuses features to generate the final cell representation.
• Figure 3: Visualization of pan-cancer classification Results. The PanFoMa can clearly separate the different cancer subtypes.
• Figure 4: Visual comparison of GRN.