Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulation

Seungjun Yi, Jaeyoung Lim, Juyong Yoon

TL;DR

ProtoMed-LLM introduces an automatic framework for evaluating LLMs on Scientific Protocol Formulation Tasks by extracting pseudocode from biology protocols using predefined lab actions and assessing outputs with Llam-Eval, using GPT-4’s pseudocode as a baseline and Llama-3 as the evaluator. The framework is complemented by the BioProt 2.0 dataset and demonstrates that predefined actions and LLAM-EVAL enable scalable, domain-aware evaluation across multiple models, including GPT variants and Cohere. Key findings show GPT-4o and Cohere excel at protocol formulation, while Llama-3 provides a cost-free evaluation backbone; the approach remains extensible to other domains requiring protocol generation. The work contributes a flexible evaluation pipeline, a robust evaluator, and a sizeable benchmark dataset to advance automatic, objective assessment of SPFT capabilities in LLMs, with practical impact on accelerating automated experimental protocol development.

Abstract

Automated generation of scientific protocols executable by robots can significantly accelerate scientific research processes. Large Language Models (LLMs) excel at Scientific Protocol Formulation Tasks (SPFT), but the evaluation of their capabilities rely on human evaluation. Here, we propose a flexible, automatic framework to evaluate LLMs' capability on SPFT: ProtoMed-LLM. This framework prompts the target model and GPT-4 to extract pseudocode from biology protocols using only predefined lab actions and evaluates the output of the target model using LLAM-EVAL, the pseudocode generated by GPT-4 serving as a baseline and Llama-3 acting as the evaluator. Our adaptable prompt-based evaluation method, LLAM-EVAL, offers significant flexibility in terms of evaluation model, material, criteria, and is free of cost. We evaluate GPT variations, Llama, Mixtral, Gemma, Cohere, and Gemini. Overall, we find that GPT and Cohere are powerful scientific protocol formulators. We also introduce BIOPROT 2.0, a dataset with biology protocols and corresponding pseudocodes, which can aid LLMs in formulation and evaluation of SPFT. Our work is extensible to assess LLMs on SPFT across various domains and other fields that require protocol generation for specific goals.

ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulation

TL;DR

ProtoMed-LLM introduces an automatic framework for evaluating LLMs on Scientific Protocol Formulation Tasks by extracting pseudocode from biology protocols using predefined lab actions and assessing outputs with Llam-Eval, using GPT-4’s pseudocode as a baseline and Llama-3 as the evaluator. The framework is complemented by the BioProt 2.0 dataset and demonstrates that predefined actions and LLAM-EVAL enable scalable, domain-aware evaluation across multiple models, including GPT variants and Cohere. Key findings show GPT-4o and Cohere excel at protocol formulation, while Llama-3 provides a cost-free evaluation backbone; the approach remains extensible to other domains requiring protocol generation. The work contributes a flexible evaluation pipeline, a robust evaluator, and a sizeable benchmark dataset to advance automatic, objective assessment of SPFT capabilities in LLMs, with practical impact on accelerating automated experimental protocol development.

Abstract

Automated generation of scientific protocols executable by robots can significantly accelerate scientific research processes. Large Language Models (LLMs) excel at Scientific Protocol Formulation Tasks (SPFT), but the evaluation of their capabilities rely on human evaluation. Here, we propose a flexible, automatic framework to evaluate LLMs' capability on SPFT: ProtoMed-LLM. This framework prompts the target model and GPT-4 to extract pseudocode from biology protocols using only predefined lab actions and evaluates the output of the target model using LLAM-EVAL, the pseudocode generated by GPT-4 serving as a baseline and Llama-3 acting as the evaluator. Our adaptable prompt-based evaluation method, LLAM-EVAL, offers significant flexibility in terms of evaluation model, material, criteria, and is free of cost. We evaluate GPT variations, Llama, Mixtral, Gemma, Cohere, and Gemini. Overall, we find that GPT and Cohere are powerful scientific protocol formulators. We also introduce BIOPROT 2.0, a dataset with biology protocols and corresponding pseudocodes, which can aid LLMs in formulation and evaluation of SPFT. Our work is extensible to assess LLMs on SPFT across various domains and other fields that require protocol generation for specific goals.
Paper Structure (22 sections, 2 equations, 2 figures, 6 tables)

This paper contains 22 sections, 2 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Data Curation of Protocols in Biology
  5. Defining Actions
  6. Prompting Pseudocode Generation
  7. Metrics and Evaluation
  8. Evaluator LLM Selection
  9. Evaluating LLMs using Llam-Eval
  10. Implementation Details
  11. Analysis
  12. Evaluator LLM Selection
  13. Evaluating LLMs on SPFT
  14. The Bioprot 2.0 Dataset
  15. Conclusion
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Overview of the ProtoMed-LLM Framework. A protocol containing a title, descriptions, step-by-step instructions, and predefined biology lab actions is given to both a target model and GPT-4 for pseudocode generation. Then, Llama-3 evaluates these outputs considering the target model's pseudocode as the prediction ($\hat{y}$) and GPT-4's as a baseline ($y$).
  • Figure 2: The ProtoMed-LLM Framework.