Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation

Yuji Naraki, Ryosuke Yamaki, Yoshikazu Ikeda, Takafumi Horie, Kotaro Yoshida, Ryotaro Shimizu, Hiroki Naganuma

TL;DR

The paper tackles noisy and costly NER data annotation by proposing a hybrid framework that uses LLMs as annotators alongside manual labeling. It introduces a label-mixing technique to handle multi-label assignments and class imbalance, implemented via a Mixup-inspired formulation with soft label distributions. Through experiments on CoNLL03 and WikiGold with 32-shot prompts and a BERT-based extractor, the approach demonstrates improved robustness and performance under limited budgets and noisy data, validating the feasibility of cost-effective high-quality NER. The work highlights practical benefits of leveraging LLMs for data annotation while mitigating their biases, and points to directions for scaling to larger datasets and more complex label schemas.

Abstract

In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is recognized as a critical technology, employed across a wide array of applications. Traditional methodologies for annotating datasets for NER models are challenged by high costs and variations in dataset quality. This research introduces a novel hybrid annotation approach that synergizes human effort with the capabilities of Large Language Models (LLMs). This approach not only aims to ameliorate the noise inherent in manual annotations, such as omissions, thereby enhancing the performance of NER models, but also achieves this in a cost-effective manner. Additionally, by employing a label mixing strategy, it addresses the issue of class imbalance encountered in LLM-based annotations. Through an analysis across multiple datasets, this method has been consistently shown to provide superior performance compared to traditional annotation methods, even under constrained budget conditions. This study illuminates the potential of leveraging LLMs to improve dataset quality, introduces a novel technique to mitigate class imbalances, and demonstrates the feasibility of achieving high-performance NER in a cost-effective way.

Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation

TL;DR

The paper tackles noisy and costly NER data annotation by proposing a hybrid framework that uses LLMs as annotators alongside manual labeling. It introduces a label-mixing technique to handle multi-label assignments and class imbalance, implemented via a Mixup-inspired formulation with soft label distributions. Through experiments on CoNLL03 and WikiGold with 32-shot prompts and a BERT-based extractor, the approach demonstrates improved robustness and performance under limited budgets and noisy data, validating the feasibility of cost-effective high-quality NER. The work highlights practical benefits of leveraging LLMs for data annotation while mitigating their biases, and points to directions for scaling to larger datasets and more complex label schemas.

Abstract

In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is recognized as a critical technology, employed across a wide array of applications. Traditional methodologies for annotating datasets for NER models are challenged by high costs and variations in dataset quality. This research introduces a novel hybrid annotation approach that synergizes human effort with the capabilities of Large Language Models (LLMs). This approach not only aims to ameliorate the noise inherent in manual annotations, such as omissions, thereby enhancing the performance of NER models, but also achieves this in a cost-effective manner. Additionally, by employing a label mixing strategy, it addresses the issue of class imbalance encountered in LLM-based annotations. Through an analysis across multiple datasets, this method has been consistently shown to provide superior performance compared to traditional annotation methods, even under constrained budget conditions. This study illuminates the potential of leveraging LLMs to improve dataset quality, introduces a novel technique to mitigate class imbalances, and demonstrates the feasibility of achieving high-performance NER in a cost-effective way.
Paper Structure (18 sections, 1 equation, 3 figures, 3 tables)

This paper contains 18 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. LLM-based Annotation
  5. Hybrid: LLM-Based and Manual Annotation
  6. Label Mixing in LLM-Based Annotation
  7. Experiments
  8. Discussion and Conclusion
  9. Ethical Considerations
  10. Limitations
  11. Details of Experiment Setup
  12. Hardware Configurations:
  13. BERT Configurations:
  14. Code Implementation
  15. Datasets
  16. ...and 3 more sections

Figures (3)

  • Figure 1: LLM-based annotation and label mixing.
  • Figure 2: Comparison of model performance when using noised and datasets annotated by Hybrid approach.
  • Figure 3: Number of annotations for each entity type in the manual, noised, and LLM-based datasets.