Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Zero to Hero: Harnessing Transformers for Biomedical Named Entity Recognition in Zero- and Few-shot Contexts

Miloš Košprdić, Nikola Prodanović, Adela Ljajić, Bojana Bašaragin, Nikola Milošević

TL;DR

This paper tackles the scarcity of annotated data in biomedical named entity recognition by transforming multi-class token classification into binary token classification and using domain-specific transformer models pre-trained on large biomedical corpora. It introduces a data transformation that pairs each entity class name with sentences, allowing zero- and few-shot learning through class-name prompts and binary labels, evaluated on 26 biomedical entity classes. Across experiments with BioBERT and PubMedBERT, the approach achieves a zero-shot macro F1 of approximately 35.4% (rising to ~40% when excluding the Dosage class) and reaches 77–87% F1 with 100 examples for many classes, outperforming several zero-shot baselines and approaching state-of-the-art in some settings. The method enables open-set NER with minimal annotation for new classes and provides publicly available models and code to facilitate adoption and further research.

Abstract

Supervised named entity recognition (NER) in the biomedical domain depends on large sets of annotated texts with the given named entities. The creation of such datasets can be time-consuming and expensive, while extraction of new entities requires additional annotation tasks and retraining the model. To address these challenges, this paper proposes a method for zero- and few-shot NER in the biomedical domain. The method is based on transforming the task of multi-class token classification into binary token classification and pre-training on a large amount of datasets and biomedical entities, which allow the model to learn semantic relations between the given and potentially novel named entity labels. We have achieved average F1 scores of 35.44% for zero-shot NER, 50.10% for one-shot NER, 69.94% for 10-shot NER, and 79.51% for 100-shot NER on 9 diverse evaluated biomedical entities with fine-tuned PubMedBERT-based model. The results demonstrate the effectiveness of the proposed method for recognizing new biomedical entities with no or limited number of examples, outperforming previous transformer-based methods, and being comparable to GPT3-based models using models with over 1000 times fewer parameters. We make models and developed code publicly available.

From Zero to Hero: Harnessing Transformers for Biomedical Named Entity Recognition in Zero- and Few-shot Contexts

TL;DR

This paper tackles the scarcity of annotated data in biomedical named entity recognition by transforming multi-class token classification into binary token classification and using domain-specific transformer models pre-trained on large biomedical corpora. It introduces a data transformation that pairs each entity class name with sentences, allowing zero- and few-shot learning through class-name prompts and binary labels, evaluated on 26 biomedical entity classes. Across experiments with BioBERT and PubMedBERT, the approach achieves a zero-shot macro F1 of approximately 35.4% (rising to ~40% when excluding the Dosage class) and reaches 77–87% F1 with 100 examples for many classes, outperforming several zero-shot baselines and approaching state-of-the-art in some settings. The method enables open-set NER with minimal annotation for new classes and provides publicly available models and code to facilitate adoption and further research.

Abstract

Supervised named entity recognition (NER) in the biomedical domain depends on large sets of annotated texts with the given named entities. The creation of such datasets can be time-consuming and expensive, while extraction of new entities requires additional annotation tasks and retraining the model. To address these challenges, this paper proposes a method for zero- and few-shot NER in the biomedical domain. The method is based on transforming the task of multi-class token classification into binary token classification and pre-training on a large amount of datasets and biomedical entities, which allow the model to learn semantic relations between the given and potentially novel named entity labels. We have achieved average F1 scores of 35.44% for zero-shot NER, 50.10% for one-shot NER, 69.94% for 10-shot NER, and 79.51% for 100-shot NER on 9 diverse evaluated biomedical entities with fine-tuned PubMedBERT-based model. The results demonstrate the effectiveness of the proposed method for recognizing new biomedical entities with no or limited number of examples, outperforming previous transformer-based methods, and being comparable to GPT3-based models using models with over 1000 times fewer parameters. We make models and developed code publicly available.
Paper Structure (13 sections, 1 equation, 3 figures, 5 tables)

This paper contains 13 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Materials and Methods
  4. Dataset
  5. Data transformation method
  6. Dataset analysis
  7. Model architecture and training method
  8. Experiments and results
  9. Pilot study: Comparison of binary and multi-class NER
  10. Main study: Zero-shot and few-shot NER using binary token classification
  11. Discussion
  12. Limitations
  13. Conclusion and future work

Figures (3)

  • Figure 1: Example of transforming one annotated sentence to the new format of input and expected output.
  • Figure 2: Model architecture (BERT refers to fine-tuned BioBERT or PubmedBERT models)
  • Figure 3: Comparison of average F1 scores for the NER task on BC5CDR corpus using five fine-tuned BERT-based models for the tasks of binary classification and multi-class NER.