BioProBench: Comprehensive Dataset and Benchmark in Biological Protocol Understanding and Reasoning

TL;DR

BioProBench addresses the need for reliable interpretation, reasoning, and generation of biological protocols by providing the first large-scale, multi-task benchmark tailored to procedural bioscience texts. It combines a raw corpus of 27K protocols with over 556K structured task instances across five tasks, plus a rigorous, domain-aware evaluation framework that includes keyword- and embedding-based metrics. Across 12 LLMs, results reveal solid basic understanding but substantial weaknesses in deep procedural reasoning, ordered generation, and safety-conscious content creation, highlighting gaps between open-source and closed-source models and between general and domain-specific models. The dataset and evaluation suite offer a benchmark-driven path toward robust protocol-aware AI systems for experimental design, automation, and safety control.

Abstract

Biological protocols are fundamental to reproducibility and safety in life science research. While large language models (LLMs) perform well on general tasks, their systematic evaluation on these highly specialized, accuracy-critical, and inherently procedural texts remains limited. In this work, we present BioProBench, the first large-scale, multi-task benchmark for biological protocol understanding and reasoning. While there are several benchmark tasks involving protocol question answering, BioProBench provides a comprehensive suite of five core tasks: Protocol Question Answering, Step Ordering, Error Correction, Protocol Generation, and Protocol Reasoning, enabling a holistic evaluation of LLMs on procedural biological texts. Built upon 27K original protocols, it yields nearly 556K high-quality structured instances. We evaluate 12 mainstream open/closed-source LLMs. Experimental results reveal that some models perform well on basic understanding tasks (e.g., \sim70% PQA-Acc., >64% ERR F1), but struggle significantly with deep reasoning and structured generation tasks like ordering and generation. Furthermore, model comparisons show diverse performance: certain open-source models approach closed-source levels on some tasks, yet bio-specific small models lag behind general LLMs, indicating limitations on complex procedural content. Overall, BioProBench, through its task design and experimental findings, systematically reveals the fundamental challenges for current LLMs in procedural knowledge understanding, deep adaptability to specific domains, reliability of structured reasoning, and handling of sophisticated precision and safety constraints, providing key directions for future AI in the field of scientific experiment automation. The code and data are available at: https://github.com/YuyangSunshine/bioprotocolbench and https://huggingface.co/datasets/BioProBench/BioProBench.

Figures (3)

• Figure 1: An overview of the BioProBench dataset and benchmark. (a) The data sources and corresponding raw data counts. (b) The biology categories of raw data counts. (c) The newly introduced classes in BioProBench. (d) The tasks and sub-tasks considered in BioProBench.
• Figure 2: Workflow of the BioProBench dataset construction. The process begins with collecting and structuring raw protocols and LLM-based data augmentation. This is followed by LLM-in-the-loop process to generate diverse task instances. Finally, tasks quality filtering through Self-Filtering mechanisms and Expert Feedback from domain specialists to ensure accuracy and relevance.
• Figure 3: Samples across each tasks from BioProBench.