SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding

TL;DR

SciHorizon-GENE presents a large-scale, gene-centric benchmark to evaluate LLMs on translating gene knowledge into functional understanding. It builds a unified, automated knowledge base from NCBI Gene, Gene Ontology, and PubMed, and generates over 540K questions across three biological scenarios and four evaluation perspectives. The study evaluates 27 models and reveals substantial heterogeneity in gene-level reasoning, with persistent gaps in faithfulness, completeness, and literature grounding, especially for low-attention genes and in multi-value attribute tasks. The findings highlight that domain-specialized models do not consistently outperform strong general-purpose LLMs and demonstrate the necessity of multi-dimensional evaluation to guide model selection and development for knowledge-enhanced biological interpretation.

Abstract

Large language models (LLMs) have shown growing promise in biomedical research, particularly for knowledge-driven interpretation tasks. However, their ability to reliably reason from gene-level knowledge to functional understanding, However, their ability to reliably reason from gene-level knowledge to functional understanding, a core requirement for knowledge-enhanced cell atlas interpretation, remains largely underexplored. To address this gap, we introduce SciHorizon-GENE, a large-scale gene-centric benchmark constructed from authoritative biological databases. The benchmark integrates curated knowledge for over 190K human genes and comprises more than 540K questions covering diverse gene-to-function reasoning scenarios relevant to cell type annotation, functional interpretation, and mechanism-oriented analysis. Motivated by behavioral patterns observed in preliminary examinations, SciHorizon-GENE evaluates LLMs along four biologically critical perspectives: research attention sensitivity, hallucination tendency, answer completeness, and literature influence, explicitly targeting failure modes that limit the safe adoption of LLMs in biological interpretation pipelines. We systematically evaluate a wide range of state-of-the-art general-purpose and biomedical LLMs, revealing substantial heterogeneity in gene-level reasoning capabilities and persistent challenges in generating faithful, complete, and literature-grounded functional interpretations. Our benchmark establishes a systematic foundation for analyzing LLM behavior at the gene scale and offers insights for model selection and development, with direct relevance to knowledge-enhanced biological interpretation.

Figures (11)

• Figure 1: Observations of LLM behavior on gene-related tasks, motivating the need for our gene-centric benchmark. (a) Model performance differs across genes with varying levels of research attention. (b) Models tend to yield hallucinated answers when responding to genes with insufficient annotation. (c) LLMs often produce partially correct but incomplete answers in multi-answer questions. (d) Model outputs vary notably depending on whether the relevant literature context is provided.
• Figure 2: The benchmark integrates curated biological databases and verified literature sources to construct gene nodes. These nodes provide a unified representation of gene attributes and support the generation of diverse, knowledge-grounded question sets. The question collections are categorized into three biological task scenarios. LLM performance is subsequently evaluated from four perspectives that reflect key behavioral patterns.
• Figure 3: PubMed reference count distribution for human genes. Dashed lines mark the mean reference counts.
• Figure 4: Model performance on three tasks for high- and low-research attention genes. All tasks are single-choice, and accuracy is used as the evaluation metric.
• Figure 5: Hallucination resistance of LLMs across Synonyms and Chromosome questions. All tasks are single-choice, and accuracy is used as the evaluation metric.