Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CellTypeAgent: Trustworthy cell type annotation with Large Language Models

Jiawen Chen, Jianghao Zhang, Huaxiu Yao, Yun Li

TL;DR

Cell type annotation in single-cell RNA-seq is essential but labor-intensive and susceptible to LLM hallucinations. The authors propose a two-stage hybrid approach: an LLM proposes top cell-type candidates from marker genes, followed by gene-expression–based verification using the CZ CELLxGENE Discover data to select the final annotation. The final decision is formalized as $c^* = \operatorname{argmax}_{c \in \mathcal{C}} \text{score}(c)$, where $\text{score}(c)$ combines the LLM-derived rank $r_c$ with expression-based ranks $\text{rank}(\sum_g e_{gc}(\mathcal{s},\tau))$ and $\text{rank}(\sum_g \rho_{gc}(\mathcal{s},\tau))$, with optional literature and gene-sum inputs. Across nine datasets, CellTypeAgent outperforms GPTCellType and CellxGene-alone baselines, and experiments with open-source Deepseek-R1 show competitive gains while addressing privacy concerns. The work demonstrates improved accuracy and reliability for scRNA-seq cell-type annotation and releases code publicly for reproducibility.

Abstract

Cell type annotation is a critical yet laborious step in single-cell RNA sequencing analysis. We present a trustworthy large language model (LLM)-agent, CellTypeAgent, which integrates LLMs with verification from relevant databases. CellTypeAgent achieves higher accuracy than existing methods while mitigating hallucinations. We evaluated CellTypeAgent across nine real datasets involving 303 cell types from 36 tissues. This combined approach holds promise for more efficient and reliable cell type annotation.

CellTypeAgent: Trustworthy cell type annotation with Large Language Models

TL;DR

Cell type annotation in single-cell RNA-seq is essential but labor-intensive and susceptible to LLM hallucinations. The authors propose a two-stage hybrid approach: an LLM proposes top cell-type candidates from marker genes, followed by gene-expression–based verification using the CZ CELLxGENE Discover data to select the final annotation. The final decision is formalized as , where combines the LLM-derived rank with expression-based ranks and , with optional literature and gene-sum inputs. Across nine datasets, CellTypeAgent outperforms GPTCellType and CellxGene-alone baselines, and experiments with open-source Deepseek-R1 show competitive gains while addressing privacy concerns. The work demonstrates improved accuracy and reliability for scRNA-seq cell-type annotation and releases code publicly for reproducibility.

Abstract

Cell type annotation is a critical yet laborious step in single-cell RNA sequencing analysis. We present a trustworthy large language model (LLM)-agent, CellTypeAgent, which integrates LLMs with verification from relevant databases. CellTypeAgent achieves higher accuracy than existing methods while mitigating hallucinations. We evaluated CellTypeAgent across nine real datasets involving 303 cell types from 36 tissues. This combined approach holds promise for more efficient and reliable cell type annotation.
Paper Structure (8 sections, 2 equations, 2 figures)

This paper contains 8 sections, 2 equations, 2 figures.

Table of Contents

  1. Main
  2. CellTypeAgent method
  3. Stage 1: LLM-based Candidates Prediction
  4. Stage 2: Gene Expression-Based Candidate Evaluation
  5. Optional literature search and gene summary
  6. Evaluation metrics
  7. Mixed cell type evaluation
  8. Evaluation dataset

Figures (2)

  • Figure 1: CellTypeAgent workflow. (a) CellTypeAgent first suggests several cell type candidates using an LLM. Then the final cell type is decided by cross-referencing with the CellxGene database. (b) A human adipose example of identified final cell type using CellTypeAgent, CellxGene, and GPTCellType.
  • Figure 2: CellTypeAgent result. (a) Average agreement score in 9 datasets. We note here that the numerical differences between our GPTCelltype scores and those reported by GPTCelltype are due to updates in the GPT-4 model. All experiments in our study were conducted using GPT models from the May to October 2024 versions. (b) Average agreement score across 9 datasets with various base LLM models. (c) Comparison of performance with different numbers of cell type candidates. (d) Comparison of performance with different numbers of marker genes. (e) Performance with mixed cell type marker genes. Each test is conducted by combining marker genes of two cell types A and B. Pure type A/B is the performance of cell type annotation using marker genes of A/B only. Mixed A&B is the performance of cell type annotation using marker genes of A and B. Avg pure A&B is the average performance of pure type A and pure type B in each test. (f) Comparison of performance with and without literature review and gene summary. (g) An example illustrating LLM's heavy reliance on inputting text.