CellVerse: Do Large Language Models Really Understand Cell Biology?

TL;DR

CellVerse introduces the first language-centric benchmark for single-cell biology by converting four multi-omics modalities into natural language and reframing CTA, DRP, and PA as QA problems. The authors systematically evaluate 14 LLMs, including both open-source and closed-source models, to assess their understanding of cell biology, finding that generalist models exhibit emerging reasoning while overall performance remains well below ideal, especially in drug response and perturbation tasks. The study reveals that specialist, task-tuned models underperform due to limited capacity and data, while scaling model size generally improves results but with notable exceptions in context utilization and task difficulty. This work provides a foundation for future research into language-driven single-cell analysis and highlights directions such as expanding data modalities, calibrating task difficulty, and exploring multilingual QA to broaden applicability and interpretability in cell biology.

Abstract

Recent studies have demonstrated the feasibility of modeling single-cell data as natural languages and the potential of leveraging powerful large language models (LLMs) for understanding cell biology. However, a comprehensive evaluation of LLMs' performance on language-driven single-cell analysis tasks still remains unexplored. Motivated by this challenge, we introduce CellVerse, a unified language-centric question-answering benchmark that integrates four types of single-cell multi-omics data and encompasses three hierarchical levels of single-cell analysis tasks: cell type annotation (cell-level), drug response prediction (drug-level), and perturbation analysis (gene-level). Going beyond this, we systematically evaluate the performance across 14 open-source and closed-source LLMs ranging from 160M to 671B on CellVerse. Remarkably, the experimental results reveal: (1) Existing specialist models (C2S-Pythia) fail to make reasonable decisions across all sub-tasks within CellVerse, while generalist models such as Qwen, Llama, GPT, and DeepSeek family models exhibit preliminary understanding capabilities within the realm of cell biology. (2) The performance of current LLMs falls short of expectations and has substantial room for improvement. Notably, in the widely studied drug response prediction task, none of the evaluated LLMs demonstrate significant performance improvement over random guessing. CellVerse offers the first large-scale empirical demonstration that significant challenges still remain in applying LLMs to cell biology. By introducing CellVerse, we lay the foundation for advancing cell biology through natural languages and hope this paradigm could facilitate next-generation single-cell analysis.

Figures (13)

• Figure 1: An illustration of traditional single-cell analysis and language-centric single-cell analysis.
• Figure 2: An overview of CellVerse. The top-left panel illustrates the composition of CellVerse, while the remaining three panels provide concrete data examples within CellVerse.
• Figure 3: Data statistics information of CellVerse.
• Figure 4: Leaderboard results on our CellVerse benchmark. The scores represent the prediction accuracy of LLMs. We include both open-source and closed-source LLMs in the evaluation.
• Figure 5: Precision score comparison of specific cell types on scRNA-seq data. All LLMs fail to predict certain challenging cell types, such as oligodendrocyte C and phagocyte.