Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data

Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, Etienne Bernard

TL;DR

The paper tackles data efficiency in NER by creating NuNER, a compact encoder trained on a large multi-domain NER dataset annotated by an LLM. Using a two-encoder contrastive pre-training objective, NuNER learns a text representation aligned with a diverse set of concepts, enabling strong few-shot transfer to downstream NER tasks. Key findings show that concept diversity and dataset size drive performance, while text diversity has limited impact when annotations come from LLMs, with NuNER achieving competitive results against much larger LLMs. The work opens access to open-source NuNER and an LLM-annotated NER dataset, supporting cost-effective, data-efficient development of task-specific foundation models for NER.

Abstract

Large Language Models (LLMs) have shown impressive abilities in data annotation, opening the way for new approaches to solve classic NLP problems. In this paper, we show how to use LLMs to create NuNER, a compact language representation model specialized in the Named Entity Recognition (NER) task. NuNER can be fine-tuned to solve downstream NER problems in a data-efficient way, outperforming similar-sized foundation models in the few-shot regime and competing with much larger LLMs. We find that the size and entity-type diversity of the pre-training dataset are key to achieving good performance. We view NuNER as a member of the broader family of task-specific foundation models, recently unlocked by LLMs.

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data

TL;DR

The paper tackles data efficiency in NER by creating NuNER, a compact encoder trained on a large multi-domain NER dataset annotated by an LLM. Using a two-encoder contrastive pre-training objective, NuNER learns a text representation aligned with a diverse set of concepts, enabling strong few-shot transfer to downstream NER tasks. Key findings show that concept diversity and dataset size drive performance, while text diversity has limited impact when annotations come from LLMs, with NuNER achieving competitive results against much larger LLMs. The work opens access to open-source NuNER and an LLM-annotated NER dataset, supporting cost-effective, data-efficient development of task-specific foundation models for NER.

Abstract

Large Language Models (LLMs) have shown impressive abilities in data annotation, opening the way for new approaches to solve classic NLP problems. In this paper, we show how to use LLMs to create NuNER, a compact language representation model specialized in the Named Entity Recognition (NER) task. NuNER can be fine-tuned to solve downstream NER problems in a data-efficient way, outperforming similar-sized foundation models in the few-shot regime and competing with much larger LLMs. We find that the size and entity-type diversity of the pre-training dataset are key to achieving good performance. We view NuNER as a member of the broader family of task-specific foundation models, recently unlocked by LLMs.
Paper Structure (20 sections, 13 figures, 11 tables)

This paper contains 20 sections, 13 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NuNER
  4. Dataset Creation
  5. Model Training
  6. Training Details
  7. Transfer Learning Performance
  8. Few-Shot with Frozen Foundation
  9. Few-Shot with TadNER on Few-NERD
  10. Ablation Studies
  11. Effect of Text Diversity
  12. Effect of Concept Diversity
  13. Effect of Dataset Size
  14. Effect of Model Size
  15. Comparison with LLMs
  16. ...and 5 more sections

Figures (13)

  • Figure 1: NuNER creation procedure. RoBERTa is further pre-trained on a subset of C4 automatically annotated by GPT-3.5. The resulting model can be fine-tuned on various downstream NER problems.
  • Figure 2: Prompt used to annotate NuNER's pre-training data.
  • Figure 3: Sentence from C4 annotated with GPT-3.5.
  • Figure 4: Frequency of each concept assigned by GPT-3.5, sorted from most to least common. We observe a heavy-tailed distribution.
  • Figure 5: NuNER's pre-tranining procedure. The text and concept encoder are separated. Their embeddings are compared to obtain probabilities.
  • ...and 8 more figures