BioPIE: A Biomedical Protocol Information Extraction Dataset for High-Reasoning-Complexity Experiment Question Answer

TL;DR

BioPIE provides a procedure-centric information extraction dataset for biomedical protocols to address high-information-density and multi-step reasoning in biomedical experiment QA. It defines a detailed KG schema (34 entity types, 21 relation types) and compiles over 10k entities and 8k relations across ID and OOD protocols, with robust annotation. The authors benchmark IE methods and develop a retrieval-augmented QA system that fuses textual evidence with protocol graphs, achieving strong performance on HID and MSR tasks and highlighting the value of structured protocol knowledge for automation. The work demonstrates the potential for protocol-level reasoning and AI-assisted laboratory workflows, while outlining avenues for graph-aware retrieval and temporal-hierarchical modeling to further enhance performance.

Abstract

Question Answer (QA) systems for biomedical experiments facilitate cross-disciplinary communication, and serve as a foundation for downstream tasks, e.g., laboratory automation. High Information Density (HID) and Multi-Step Reasoning (MSR) pose unique challenges for biomedical experimental QA. While extracting structured knowledge, e.g., Knowledge Graphs (KGs), can substantially benefit biomedical experimental QA. Existing biomedical datasets focus on general or coarsegrained knowledge and thus fail to support the fine-grained experimental reasoning demanded by HID and MSR. To address this gap, we introduce Biomedical Protocol Information Extraction Dataset (BioPIE), a dataset that provides procedure-centric KGs of experimental entities, actions, and relations at a scale that supports reasoning over biomedical experiments across protocols. We evaluate information extraction methods on BioPIE, and implement a QA system that leverages BioPIE, showcasing performance gains on test, HID, and MSR question sets, showing that the structured experimental knowledge in BioPIE underpins both AI-assisted and more autonomous biomedical experimentation.

Figures (7)

• Figure 1: biopie enhances complex biomedical protocol understanding.(A) The kgkg in biopie provide fine-grained structural representations of experimental steps (, temperature, duration, and execution order), resulting in high information density, and enable multi-step reasoning by integrating sentence-level context with graph-structured knowledge. (B) Existing information extraction datasets involve a trade-off: general datasets lack biomedical knowledge, while domain-specific datasets may not generalize across diverse experiments.
• Figure 2: Illustration of biopie.(A) An annotated example of a biomedical experimental protocol for plasmid DNA preparation, illustrating how diverse laboratory operations are decomposed into structured procedural entities and relations under our annotation schema, independent of domain-specific biological semantics. (B) Statistics of entity types and relation types in the biopie dataset. (C) Representative entity and relation labels in our annotation scheme, with definitions and examples.
• Figure 3: Effects of Settings on ie Methods.(A) Impact of the number of retrieval on validation set for Rel+ F1 score. (B) Performance trends of PL-Marker under varying training-protocol proportions on the id test set.
• Figure 4: Pipeline of proposed qa system.
• Figure 5: Effects of the number of retrieval on qa systems performance for validation set.