Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanism

TL;DR

The paper tackles the challenge of generating reproducible, executable bio-experimental protocols by building a specialized ecosystem: the SciRecipe dataset, the Sketch-and-Fill reasoning paradigm, and the SCORE structured reward mechanism, all integrated into a Knowledge-to-Action training pipeline for the Thoth protocol generator. This approach shifts protocol generation from plain text to structured, verifiable reasoning and evaluation, enabling both improved executability and semantic fidelity. Across protocol-generation and broader biomedical reasoning benchmarks, Thoth delivers state-of-the-art performance and demonstrates robust generalization, supported by ablation studies that validate the value of structured reasoning, gating mechanisms, and staged learning. The work advances reproducible science by bridging knowledge with actionable laboratory procedures and provides open resources to enable reproducibility and further development in automated protocol design.

Abstract

The foundation of reproducible science lies in protocols that are precise, logically ordered, and executable. The autonomous generation of these protocols through natural language queries could greatly improve the efficiency of the reproduction process. However, current leading large language models (LLMs) often generate incomplete or inconsistent protocols, limiting their utility. To address this limitation, we first introduce SciRecipe, a large-scale dataset of over 12K structured protocols spanning 27 biological subfields and encompassing both comprehension and problem-solving tasks. To further improve protocol generation, we propose the "Sketch-and-Fill" paradigm, which separates analysis, structuring, and expression to ensure each step is explicit and verifiable. Complementing this, the structured component-based reward mechanism evaluates step granularity, action order, and semantic fidelity, aligning model optimization with experimental reliability. Building on these components, we develop Thoth, trained through a staged Knowledge-to-Action process that progresses from knowledge acquisition to operational reasoning and ultimately to robust, executable protocol generation. Across multiple benchmarks, Thoth consistently surpasses both proprietary and open-source LLMs, achieving significant improvements in step alignment, logical sequencing, and semantic accuracy. Our approach paves the way for reliable scientific assistants that bridge knowledge with experimental execution. All data, code, and models will be released publicly.

Figures (25)

• Figure 1: (a) Thoth’s performance advantages over other models, shown through qualitative evaluation (left) and quantitative evaluation (right). (b) The construction pipeline of the SciRecipe dataset.
• Figure 2: Illustration of the SCORE mechanism and the “Sketch-and-Fill” reasoning paradigm. Three representative rollouts are subjected to a concise analysis across step scale, order consistency, and semantic consistency, while detailed computations are provided in Section \ref{['Reward Design']}.
• Figure 3: Results on other scientific benchmarks. Slash shading denotes baseline of Thoth models.
• Figure 4: Ablation studies on reasoning paradigm. Performance differences with/without Sketch-and-Fill (SaF) are shown in the top panel, and a visual comparison is provided in the bottom panel.
• Figure 5: Ablation studies on the training strategy. Left and middle: rightmost $\Delta$ marks extreme values across strategies. Right: Thoth’s training trajectory across stages.