MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics

TL;DR

MetaBench introduces the first metabolomics-specific multi-task benchmark to evaluate LLM capabilities across five core tasks: knowledge recall, pathway understanding, cross-database grounding, natural-language reasoning, and research description generation. By curating 8,100 samples from HMDB, KEGG, PathBank, MetaKG, and MetaboLights and evaluating 25 state-of-the-art models, the study reveals strong performance on generation tasks but a critical bottleneck in cross-database identifier grounding, which remains near-zero accuracy without retrieval augmentation. Grounding bottlenecks and long-tail metabolite coverage emerge as fundamental challenges, suggesting that scaling alone cannot address domain-specific integration and disambiguation. MetaBench provides a extensible framework for standardized assessment and targeted improvements, enabling reliable AI-assisted metabolomics research and guiding future domain-aware modeling and data augmentation efforts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities on general text; however, their proficiency in specialized scientific domains that require deep, interconnected knowledge remains largely uncharacterized. Metabolomics presents unique challenges with its complex biochemical pathways, heterogeneous identifier systems, and fragmented databases. To systematically evaluate LLM capabilities in this domain, we introduce MetaBench, the first benchmark for metabolomics assessment. Curated from authoritative public resources, MetaBench evaluates five capabilities essential for metabolomics research: knowledge, understanding, grounding, reasoning, and research. Our evaluation of 25 open- and closed-source LLMs reveals distinct performance patterns across metabolomics tasks: while models perform well on text generation tasks, cross-database identifier grounding remains challenging even with retrieval augmentation. Model performance also decreases on long-tail metabolites with sparse annotations. With MetaBench, we provide essential infrastructure for developing and evaluating metabolomics AI systems, enabling systematic progress toward reliable computational tools for metabolomics research.

Figures (6)

• Figure 1: MetaBench construction and task design. MetaBench integrates data from multiple datasets to assess five capabilities: knowledge, understanding, grounding, reasoning, and research.
• Figure 2: The statistics of MetaBench datasets.
• Figure 3: Parameter scaling trends in MetaBench. Average performance increases roughly log-linearly with model size, with consistent family-wise improvements.
• Figure 4: LLMs consistently fail in metabolite identifier grounding. 11 LLMs were asked to retrieve KEGG ID for HMDB0004148. None produced the correct answer. In contrast, when augmented with web search API, all successfully provided the correct mapping.
• Figure 5: Without search capabilities (blue bars), all models achieve less than 1% accuracy on cross-database metabolite mapping. Enabling web search (orange bars) improves accuracy by 40–150×, though performance remains below 41% even for the best model (GPT-5), indicating that retrieval alone is insufficient to resolve the grounding bottleneck.