Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain

Mingchen Li, Rui Zhang

TL;DR

The paper tackles the critical problem of achieving near-perfect few-shot medical NER with limited labeled data. It conducts a comprehensive benchmark of 16 NER models including SLMs and LLMs on BC5CDR and NCBI, and introduces the RT (Retrieving and Thinking) framework that combines example retrieval with stepwise reasoning to enhance entity recognition. Key contributions include a thorough cross-model evaluation, the novel RT methodology with ablations, and detailed analyses of prompt strategies, data quality, and domain transfer effects. The findings indicate that LLMs outperform SLMs when provided with high-quality instructions and reasoning, but challenges such as misidentification and template errors persist; domain-specific pre-training and dataset-aware example selection are crucial for improvement. Overall, the work provides practical guidance for deploying medical few-shot NER in data-scarce settings and presents RT as a concrete method to push toward higher, yet not perfect, accuracy in real-world scenarios.

Abstract

Recent advancements in language models (LMs) have led to the emergence of powerful models such as Small LMs (e.g., T5) and Large LMs (e.g., GPT-4). These models have demonstrated exceptional capabilities across a wide range of tasks, such as name entity recognition (NER) in the general domain. (We define SLMs as pre-trained models with fewer parameters compared to models like GPT-3/3.5/4, such as T5, BERT, and others.) Nevertheless, their efficacy in the medical section remains uncertain and the performance of medical NER always needs high accuracy because of the particularity of the field. This paper aims to provide a thorough investigation to compare the performance of LMs in medical few-shot NER and answer How far is LMs from 100\% Few-shot NER in Medical Domain, and moreover to explore an effective entity recognizer to help improve the NER performance. Based on our extensive experiments conducted on 16 NER models spanning from 2018 to 2023, our findings clearly indicate that LLMs outperform SLMs in few-shot medical NER tasks, given the presence of suitable examples and appropriate logical frameworks. Despite the overall superiority of LLMs in few-shot medical NER tasks, it is important to note that they still encounter some challenges, such as misidentification, wrong template prediction, etc. Building on previous findings, we introduce a simple and effective method called \textsc{RT} (Retrieving and Thinking), which serves as retrievers, finding relevant examples, and as thinkers, employing a step-by-step reasoning process. Experimental results show that our proposed \textsc{RT} framework significantly outperforms the strong open baselines on the two open medical benchmark datasets

How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain

TL;DR

The paper tackles the critical problem of achieving near-perfect few-shot medical NER with limited labeled data. It conducts a comprehensive benchmark of 16 NER models including SLMs and LLMs on BC5CDR and NCBI, and introduces the RT (Retrieving and Thinking) framework that combines example retrieval with stepwise reasoning to enhance entity recognition. Key contributions include a thorough cross-model evaluation, the novel RT methodology with ablations, and detailed analyses of prompt strategies, data quality, and domain transfer effects. The findings indicate that LLMs outperform SLMs when provided with high-quality instructions and reasoning, but challenges such as misidentification and template errors persist; domain-specific pre-training and dataset-aware example selection are crucial for improvement. Overall, the work provides practical guidance for deploying medical few-shot NER in data-scarce settings and presents RT as a concrete method to push toward higher, yet not perfect, accuracy in real-world scenarios.

Abstract

Recent advancements in language models (LMs) have led to the emergence of powerful models such as Small LMs (e.g., T5) and Large LMs (e.g., GPT-4). These models have demonstrated exceptional capabilities across a wide range of tasks, such as name entity recognition (NER) in the general domain. (We define SLMs as pre-trained models with fewer parameters compared to models like GPT-3/3.5/4, such as T5, BERT, and others.) Nevertheless, their efficacy in the medical section remains uncertain and the performance of medical NER always needs high accuracy because of the particularity of the field. This paper aims to provide a thorough investigation to compare the performance of LMs in medical few-shot NER and answer How far is LMs from 100\% Few-shot NER in Medical Domain, and moreover to explore an effective entity recognizer to help improve the NER performance. Based on our extensive experiments conducted on 16 NER models spanning from 2018 to 2023, our findings clearly indicate that LLMs outperform SLMs in few-shot medical NER tasks, given the presence of suitable examples and appropriate logical frameworks. Despite the overall superiority of LLMs in few-shot medical NER tasks, it is important to note that they still encounter some challenges, such as misidentification, wrong template prediction, etc. Building on previous findings, we introduce a simple and effective method called \textsc{RT} (Retrieving and Thinking), which serves as retrievers, finding relevant examples, and as thinkers, employing a step-by-step reasoning process. Experimental results show that our proposed \textsc{RT} framework significantly outperforms the strong open baselines on the two open medical benchmark datasets
Paper Structure (23 sections, 5 figures, 5 tables)

This paper contains 23 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Small Language Models (SLMs) in Medical Few-Shot NER
  4. Large Language Models (LLMs) in Medical Few-Shot NER
  5. Small LMs v.s. Large LMs
  6. Problem Statement and Setups
  7. Few-Shot NER
  8. Setups
  9. Dataset
  10. SLMs, LLMs and Evaluation Metrics
  11. Small Language Models
  12. Large Language Models
  13. Comparison Results
  14. More Analysis on LLMs
  15. Positive and Negative Samples in the Chain-of-Thought
  16. ...and 8 more sections

Figures (5)

  • Figure 1: LLM performance on without (w/o) mask operation and with (w/) mask operation
  • Figure 2: The output format of different ICL methods
  • Figure 3: LLM performance on without (w/o) negative samples and with (w/) negative samples
  • Figure 4: (a) refers to the comparison of the chain of thought and tree of thought. (b) The LLM performance of the chain of thought and tree of thought.
  • Figure 5: The overview about our framework RT.