Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models

Tingyu Xie, Qi Li, Yan Zhang, Zuozhu Liu, Hongwei Wang

TL;DR

This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs.

Abstract

Exploring the application of powerful large language models (LLMs) on the named entity recognition (NER) task has drawn much attention recently. This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs. First, we use the LLM to make predictions on the unlabeled corpus using self-consistency and obtain a self-annotated dataset. Second, we explore various strategies to select reliable annotations to form a reliable self-annotated dataset. Finally, for each test input, we retrieve demonstrations from the reliable self-annotated dataset and perform inference via in-context learning. Experiments on four benchmarks show substantial performance improvements achieved by our framework. Through comprehensive experimental analysis, we find that increasing the size of unlabeled corpus or iterations of self-improving does not guarantee further improvement, but the performance might be boosted via more advanced strategies for reliable annotation selection. Code and data are publicly available at https://github.com/Emma1066/Self-Improve-Zero-Shot-NER

Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models

TL;DR

This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs.

Abstract

Exploring the application of powerful large language models (LLMs) on the named entity recognition (NER) task has drawn much attention recently. This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs. First, we use the LLM to make predictions on the unlabeled corpus using self-consistency and obtain a self-annotated dataset. Second, we explore various strategies to select reliable annotations to form a reliable self-annotated dataset. Finally, for each test input, we retrieve demonstrations from the reliable self-annotated dataset and perform inference via in-context learning. Experiments on four benchmarks show substantial performance improvements achieved by our framework. Through comprehensive experimental analysis, we find that increasing the size of unlabeled corpus or iterations of self-improving does not guarantee further improvement, but the performance might be boosted via more advanced strategies for reliable annotation selection. Code and data are publicly available at https://github.com/Emma1066/Self-Improve-Zero-Shot-NER
Paper Structure (20 sections, 6 figures, 12 tables)

This paper contains 20 sections, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Zero-Shot NER with Self-Improving
  3. Step 1: Zero-Shot Self-Annotating
  4. Step 2: Reliable Annotation Selection
  5. Step 3: Inference with Self-Annotated Demonstration
  6. Experiment
  7. Setup
  8. Results
  9. Analysis
  10. Related Work
  11. Conclusion
  12. Dataset Statistics
  13. Results on Additional Benchmarks
  14. Results on Other Embedding Models
  15. Results on Various Number of Demonstrations
  16. ...and 5 more sections

Figures (6)

  • Figure 1: The overview of the proposed self-improving framework for zero-shot NER with LLM.
  • Figure 2: Results of increasing the size of unlabeled dataset. Vertical axes represent F1 scores. Ours refers to the combination of two-stage majority voting and diverse nearest with SC ranking. Increasing unlabeled data does not guarantee performance gains.
  • Figure 3: Increasing the iterations of self-improving does not guarantee performance improvements.
  • Figure 4: Kernel density estimation for SC scores. Vertical axes represent density, horizontal axes represent SC scores.
  • Figure 5: The pipeline of iterative self-improving.
  • ...and 1 more figures