Benchmarking AI scientists for omics data driven biological discovery

TL;DR

This work introduces BAISBench, a data-driven benchmark to evaluate AI scientists on realistic omics data through two tasks: BAIS-DPTA, which tests end-to-end single-cell data processing and cell type annotation using a hierarchical uHAF framework, and BAIS-SD, which assesses data-driven biological reasoning via 193 questions derived from 41 published single-cell studies. Five representative AI scientists and six human bioinformaticians are used to benchmark performance, revealing that AI systems can reliably execute standard preprocessing workflows, yet still lag in nuanced biological interpretation compared to humans; in discovery tasks, architectures like STELLA and stronger base LLMs enable performance approaching human experts, with significant influence from token budgets. The results highlight complementary strengths between AI scientists and human researchers, suggesting AI systems are best suited as assistants that automate procedural analyses and enable systematic data exploration, while humans provide domain intuition and contextual interpretation. BAISBench thus offers a practical framework to diagnose bottlenecks, guide the development of more capable AI scientists, and facilitate collaboration between AI and human researchers in data-driven biology.

Abstract

Recent advances in large language models have enabled the emergence of AI scientists that aim to autonomously analyze biological data and assist scientific discovery. Despite rapid progress, it remains unclear to what extent these systems can extract meaningful biological insights from real experimental data. Existing benchmarks either evaluate reasoning in the absence of data or focus on predefined analytical outputs, failing to reflect realistic, data-driven biological research. Here, we introduce BAISBench (Biological AI Scientist Benchmark), a benchmark for evaluating AI scientists on real single-cell transcriptomic datasets. BAISBench comprises two tasks: cell type annotation across 15 expert-labeled datasets, and scientific discovery through 193 multiple-choice questions derived from biological conclusions reported in 41 published single-cell studies. We evaluated several representative AI scientists using BAISBench and, to provide a human performance baseline, invited six graduate-level bioinformaticians to collectively complete the same tasks. The results show that while current AI scientists fall short of fully autonomous biological discovery, they already demonstrate substantial potential in supporting data-driven biological research. These results position BAISBench as a practical benchmark for characterizing the current capabilities and limitations of AI scientists in biological research. We expect BAISBench to serve as a practical evaluation framework for guiding the development of more capable AI scientists and for helping biologists identify AI systems that can effectively support real-world research workflows. The BAISBench can be found at: https://github.com/EperLuo/BAISBench, https://huggingface.co/datasets/EperLuo/BaisBench.

Figures (6)

• Figure 1: (A) Overview of BAISBench. (B) Construction of BAIS-DPTA: multi-organ single-cell transcriptomic datasets were curated and expert-annotated to support hierarchical cell type evaluation using uHAF. (C) Construction of BAIS-SD: published single-cell studies and corresponding datasets were curated to derive data-driven multiple-choice questions, which were completed by both AI scientists and graduate-level bioinformaticians.
• Figure 2: Pipeline of the BAIS-DPTA task. The AI scientist is provided with a single-cell gene expression dataset from a specific organ and is required to perform cell type annotation using its own chosen method. The predicted annotations are then evaluated using a hierarchical scoring metric based on the uHAF cell type tree, which quantifies performance according to the granularity and correctness of the predictions.
• Figure 3: (A) Pipeline of the BAIS-SD task. The AI scientist is provided with background information and a corresponding single-cell dataset, and is required to answer multiple-choice questions by performing data analysis and biological reasoning. Its answers are then compared against the ground truth derived from published literature. (B) Distribution of question categories in the BAIS-SD multiple-choice question set.
• Figure 4: The cell type annotation accuracy of different AI models in the BAIS-DPTA task. (A) Overall results. (B) Results on different organs. The results of AutoBA are not shown in (B) as they are all zero.
• Figure 5: The relationship of AI scientist performance and LLM token consumption in (A) BAIS-DPTA, (B) BAIS-SD using different AI scientists, and (C) BAIS-SD using Biomni with diferent base LLM models.