Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training

TL;DR

OKR-CELL presents an open-world, knowledge-augmented single-cell foundation model that fuses an scRNA-seq encoder with a text encoder through a cross-modal projector. It introduces a retrieval-augmented generation (RAG) pipeline to enrich cell descriptions with open-world knowledge, and a Cross-modal Robust Alignment (CRA) objective that handles sample reliability, curriculum learning, and momentum-based memory banks to combat noise. Pretrained on 32 million cell–text pairs (SCxGEN-32M), OKR-CELL achieves state-of-the-art results across clustering, cell-type annotation (including few-shot and zero-shot), batch correction, and cross-modal retrieval, demonstrating robustness to noisy data and strong generalization to unseen cell types. The work highlights the potential of integrating LLM-derived textual knowledge with robust cross-modal alignment to advance multi-modal single-cell analysis and lays the foundation for future multimodal, interactive exploration of cellular data.

Abstract

Recent advancements in single-cell multi-omics, particularly RNA-seq, have provided profound insights into cellular heterogeneity and gene regulation. While pre-trained language model (PLM) paradigm based single-cell foundation models have shown promise, they remain constrained by insufficient integration of in-depth individual profiles and neglecting the influence of noise within multi-modal data. To address both issues, we propose an Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model (OKR-CELL). It is built based on a cross-modal Cell-Language pre-training framework, which comprises two key innovations: (1) leveraging Large Language Models (LLMs) based workflow with retrieval-augmented generation (RAG) enriches cell textual descriptions using open-world knowledge; (2) devising a Cross-modal Robust Alignment (CRA) objective that incorporates sample reliability assessment, curriculum learning, and coupled momentum contrastive learning to strengthen the model's resistance to noisy data. After pretraining on 32M cell-text pairs, OKR-CELL obtains cutting-edge results across 6 evaluation tasks. Beyond standard benchmarks such as cell clustering, cell-type annotation, batch-effect correction, and few-shot annotation, the model also demonstrates superior performance in broader multi-modal applications, including zero-shot cell-type annotation and bidirectional cell-text retrieval.

Figures (6)

• Figure 1: (A) The schematic overview of the OKR-CELL method. (B) The illustration of several downstream tasks implemented via OKR-CELL, including cell clustering, batch affect correlation, cell-type annotation and cross-modal retrieval.
• Figure 2: (A) Summary table of ARI and AMI scores for cell clustering across three datasets (blood, kidney, and hPancreas). (B) Bar plot of AvgBIO scores across four datasets (blood, kidney, hPancreas and hpbmc). (C) T-SNE plots of cell embeddings generated by different methods on blood and hPancreas datasets. (D) Bar plot of AvgBIO and AvgBatch scores of batch effect correction by different methods on hPancreas dataset. (E) Bar plot of AvgBIO and AvgBatch scores of batch effect correction by different methods on hpbmc reference dataset. (F) Radar chart of 8 cretria evaluating the effectiveness of batch effect correction by different methods on hPancreas reference dataset.
• Figure 3: (A) T-SNE visualization plots of cell embeddings generated by different methods on eye, hpbmc and small-intestine reference datasets colored by ground-truth cell types. (B) Normalized confusion matrix by cell types on kidney and small-intestine reference datasets by OKR-CELL method. (C) Bar plot of accuracy and f1-score for cell type annotation by different methods on small-intestine, spleen and hpbmc reference datasets. (D) Bar plot of accuracy and f1-score for few-shot cell type annotation different methods on zheng68k and Great-Apes reference datasets. (E) Bar plot of accuracy and f1-score for zero-shot cell type annotation different methods on eye, prostate-gland and Great-Apes reference datasets. (F) Sankey diagram visualizes the correspondence between predicted labels and reference annotations for zero-shot cell type annotation by LangCell and OKR-CELL methods on prostate-gland reference dataset.
• Figure 4: (A) Curve plot of cell type annotation performance of scCLIP-GPT and OKR-CELL under varying gene dropout rates in input scRNA-seq data. (B) Bar plot of traditional cell type annotation performance by scCLIP-GPT and OKR-CELL trained under noisy cell-text data, where the obtained methods denoted by scCLIP-GPT(noisy) and OKR-CELL(noisy). (C) Bar plot of zero-shot cell type annotation performance by scCLIP-GPT(noisy) and OKR-CELL(noisy) along with original scCLIP-GPT and OKR-CELL.
• Figure 5: (A) Bar plot of bidirectional cross-modal retrieval performance (R@1,5,10) by different methods on our proposed SCxGEN-CT5K dataset. (B) T-SNE visualization plots of cell embeddings and textual embeddings of their corresponding cell types generated by different methods on eye and Great Apes reference datasets colored by ground-truth cell types. (C) Results visualization of cell based text retrieval on SCxGEN-CT5K dataset. The ground-truth and non ground-truth descriptions are marked in green and red, respectively. Note that, in addition to the original cell’s paired cell-type textual description, any text descriptions corresponding to other subtypes under the same parent cell type are also marked in green. Samples with correct broad-type but incorrect subtype are considered partially correct and are highlighted in purple.