Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions

Yichong Zhao, Susumu Goto

TL;DR

This work examines whether frontier LLMs can replace human annotators in biomedical text mining by diagnosing failure patterns and creating a targeted prompt-engineering pipeline. It introduces dynamic few-shot prompting, two-step reasoning, and instruction retrieval from annotation guidelines to align LLM outputs with complex biomedical schemas, plus a retrieval-guided workflow for guideline chunks. GPT-4o, equipped with these techniques, can approach or exceed SOTA performance with minimal manually labeled data, and the authors demonstrate a practical distillation path by fine-tuning BioLinkBERT on LLM-annotated data to achieve competitive results. The study discusses production feasibility, data-contamination risks, and future directions such as automated guideline querying and retriever-based guidance to further reduce annotation effort in real-world deployments.

Abstract

Multiple previous studies have reported suboptimal performance of LLMs in biomedical text mining. By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora: (1) LLMs fail to learn implicit dataset-specific nuances from supervised data, (2) The common formatting requirements of discriminative tasks limit the reasoning capabilities of LLMs particularly for LLMs that lack test-time compute, and (3) LLMs struggle to adhere to annotation guidelines and match exact schemas, which hinders their ability to understand detailed annotation requirements which is essential in biomedical annotation workflow. We experimented with prompt engineering techniques targeted to the above issues, and developed a pipeline that dynamically extracts instructions from annotation guidelines. Our results show that frontier LLMs can approach or surpass the performance of SOTA BERT-based models with minimal reliance on manually annotated data and without fine-tuning. Furthermore, we performed model distillation on a closed-source LLM, demonstrating that a BERT model trained exclusively on synthetic data annotated by LLMs can also achieve a practical performance. Based on these findings, we explored the feasibility of partially replacing manual annotation with LLMs in production scenarios for biomedical text mining.

Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions

TL;DR

This work examines whether frontier LLMs can replace human annotators in biomedical text mining by diagnosing failure patterns and creating a targeted prompt-engineering pipeline. It introduces dynamic few-shot prompting, two-step reasoning, and instruction retrieval from annotation guidelines to align LLM outputs with complex biomedical schemas, plus a retrieval-guided workflow for guideline chunks. GPT-4o, equipped with these techniques, can approach or exceed SOTA performance with minimal manually labeled data, and the authors demonstrate a practical distillation path by fine-tuning BioLinkBERT on LLM-annotated data to achieve competitive results. The study discusses production feasibility, data-contamination risks, and future directions such as automated guideline querying and retriever-based guidance to further reduce annotation effort in real-world deployments.

Abstract

Multiple previous studies have reported suboptimal performance of LLMs in biomedical text mining. By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora: (1) LLMs fail to learn implicit dataset-specific nuances from supervised data, (2) The common formatting requirements of discriminative tasks limit the reasoning capabilities of LLMs particularly for LLMs that lack test-time compute, and (3) LLMs struggle to adhere to annotation guidelines and match exact schemas, which hinders their ability to understand detailed annotation requirements which is essential in biomedical annotation workflow. We experimented with prompt engineering techniques targeted to the above issues, and developed a pipeline that dynamically extracts instructions from annotation guidelines. Our results show that frontier LLMs can approach or surpass the performance of SOTA BERT-based models with minimal reliance on manually annotated data and without fine-tuning. Furthermore, we performed model distillation on a closed-source LLM, demonstrating that a BERT model trained exclusively on synthetic data annotated by LLMs can also achieve a practical performance. Based on these findings, we explored the feasibility of partially replacing manual annotation with LLMs in production scenarios for biomedical text mining.

Paper Structure

This paper contains 25 sections, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background: Biomedical Text Mining and the Role of Language Models
  3. Contribution of Our Work
  4. Problem Analysis: LLM in Biomedical Text Mining
  5. Biomedical Tasks and Datasets
  6. Challenges Faced by LLM
  7. Methodology
  8. Prompt Engineering
  9. Dynamic Few-shot Prompting
  10. Strategies for Structured Output
  11. Two-step Inference to Maintain Both Reasoning and Structured Outputs
  12. Automatic Prompt Optimization
  13. Instruction Retrieval from Annotation Guideline
  14. Model Distillation
  15. Results and Discussion
  16. ...and 10 more sections

Figures (9)

  • Figure 1: The Reliance of Biomedical Text Mining Tasks on Human Annotators
  • Figure 2: Models fine-tuned via supervised learning exhibit label distributions that align more closely with the training set and ground truth.
  • Figure 4: Workflow of the Dynamic Few-shot Prompting
  • Figure 5: Workflow of the Two-step Inference to Maintain Both Reasoning Step and Structured Outputs
  • Figure 6: Automatic Prompt Optimization Using Pseudo "Gradient Descent"
  • ...and 4 more figures