Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fighting Against the Repetitive Training and Sample Dependency Problem in Few-shot Named Entity Recognition

Chang Tian, Wenpeng Yin, Dan Li, Marie-Francine Moens

TL;DR

This work tackles two persistent challenges in few-shot NER: repetitive training of basic span features in span detectors and the sample dependency problem in metric-based entity typing. It introduces SMCS, a pipeline that initializes span detection with a Steppingstone Span Detector pre-trained on Wikipedia and leverages machine common sense to construct entity-type referents from GPT-3.5 definitions, reducing dependence on scarce per-type samples. Through extensive experiments on Few-NERD and Cross-Dataset benchmarks, SMCS achieves faster convergence and superior or competitive performance relative to strong baselines, including ChatGPT in fine-grained scenarios. The study provides practical resources, including the steppingstone detector and referent-generation workflow, to support domain adaptation and low-resource NER tasks in industry and research.

Abstract

Few-shot named entity recognition (NER) systems recognize entities using a few labeled training examples. The general pipeline consists of a span detector to identify entity spans in text and an entity-type classifier to assign types to entities. Current span detectors rely on extensive manual labeling to guide training. Almost every span detector requires initial training on basic span features followed by adaptation to task-specific features. This process leads to repetitive training of the basic span features among span detectors. Additionally, metric-based entity-type classifiers, such as prototypical networks, typically employ a specific metric that gauges the distance between the query sample and entity-type referents, ultimately assigning the most probable entity type to the query sample. However, these classifiers encounter the sample dependency problem, primarily stemming from the limited samples available for each entity-type referent. To address these challenges, we proposed an improved few-shot NER pipeline. First, we introduce a steppingstone span detector that is pre-trained on open-domain Wikipedia data. It can be used to initialize the pipeline span detector to reduce the repetitive training of basic features. Second, we leverage a large language model (LLM) to set reliable entity-type referents, eliminating reliance on few-shot samples of each type. Our model exhibits superior performance with fewer training steps and human-labeled data compared with baselines, as demonstrated through extensive experiments on various datasets. Particularly in fine-grained few-shot NER settings, our model outperforms strong baselines, including ChatGPT. We will publicly release the code, datasets, LLM outputs, and model checkpoints.

Fighting Against the Repetitive Training and Sample Dependency Problem in Few-shot Named Entity Recognition

TL;DR

This work tackles two persistent challenges in few-shot NER: repetitive training of basic span features in span detectors and the sample dependency problem in metric-based entity typing. It introduces SMCS, a pipeline that initializes span detection with a Steppingstone Span Detector pre-trained on Wikipedia and leverages machine common sense to construct entity-type referents from GPT-3.5 definitions, reducing dependence on scarce per-type samples. Through extensive experiments on Few-NERD and Cross-Dataset benchmarks, SMCS achieves faster convergence and superior or competitive performance relative to strong baselines, including ChatGPT in fine-grained scenarios. The study provides practical resources, including the steppingstone detector and referent-generation workflow, to support domain adaptation and low-resource NER tasks in industry and research.

Abstract

Few-shot named entity recognition (NER) systems recognize entities using a few labeled training examples. The general pipeline consists of a span detector to identify entity spans in text and an entity-type classifier to assign types to entities. Current span detectors rely on extensive manual labeling to guide training. Almost every span detector requires initial training on basic span features followed by adaptation to task-specific features. This process leads to repetitive training of the basic span features among span detectors. Additionally, metric-based entity-type classifiers, such as prototypical networks, typically employ a specific metric that gauges the distance between the query sample and entity-type referents, ultimately assigning the most probable entity type to the query sample. However, these classifiers encounter the sample dependency problem, primarily stemming from the limited samples available for each entity-type referent. To address these challenges, we proposed an improved few-shot NER pipeline. First, we introduce a steppingstone span detector that is pre-trained on open-domain Wikipedia data. It can be used to initialize the pipeline span detector to reduce the repetitive training of basic features. Second, we leverage a large language model (LLM) to set reliable entity-type referents, eliminating reliance on few-shot samples of each type. Our model exhibits superior performance with fewer training steps and human-labeled data compared with baselines, as demonstrated through extensive experiments on various datasets. Particularly in fine-grained few-shot NER settings, our model outperforms strong baselines, including ChatGPT. We will publicly release the code, datasets, LLM outputs, and model checkpoints.
Paper Structure (28 sections, 13 equations, 9 figures, 12 tables)

This paper contains 28 sections, 13 equations, 9 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Task Formulation
  4. Model
  5. Entity Span Detector
  6. Steppingstone Span Detector
  7. Domain Adaptation of Span Detector
  8. Entity Classification
  9. Machine Common Sense
  10. Entity Span Classification
  11. Experiment
  12. Experiment Setup
  13. Datasets and Baselines
  14. Evaluation Metrics and Implementation Details
  15. Research Questions
  16. ...and 13 more sections

Figures (9)

  • Figure 1: An example of token-wise label dependency. The label dependency is shown with transition probabilities between entity labels; for example, the probability from I to O is 0.3 in the left table. This figure is from the work 08, which assumes that an abstract transition probability is evenly split into related target transitions. However, this does not match reality.
  • Figure 2: 2-way 2-shot NER task. Model detects new entities using two shots per target type. Colors represent different entity types.
  • Figure 3: A sample dependency problem in the 1-shot case. The yellow circle represents a query sample, while the red circle and purple triangle are 1-shot samples from classes A and B, respectively. Despite belonging to class A, the query sample (yellow circle) is misclassified as class B due to the absence of a closer reference sample from its own class. Instead, the nearest available reference is a purple triangle from class B, causing the query sample's misclassification. This misclassification stems from its sample dependency on the purple sample from class B.
  • Figure 4: SMCS model framework. In step 1, the Span Detector is initialized with a Pre-trained Span Detector and learns in the source domain. It then finetunes on the target domain to detect spans. In step 2, a pre-trained language model like GPT-3.5 generates entity type definitions, encoding them as type referents in a vector space. The predicted span's embedding from step 1 is compared to the type referents to determine its entity type.
  • Figure 5: Automatic annotation of Wikipedia web data.
  • ...and 4 more figures