Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Qi Zhan, Yile Wang, Hui Huang

TL;DR

This work systematically assesses generative NER using open-source LLMs across flat and nested NER tasks, focusing on output formats, memorization, and general capability preservation after instruction tuning. It finds that parameter-efficient fine-tuning with structured inline formats can match encoder-based NER and outperform GPT-3, with performance driven by instruction-following rather than memorization. The study also shows that instruction tuning largely preserves general multi-task abilities, and in some cases improves tasks requiring entity understanding like DROP. Overall, generative NER with LLMs emerges as a viable, user-friendly alternative to traditional methods, with broad practical impact for information extraction in diverse domains.

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass closed-source LLMs like GPT-3; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

TL;DR

This work systematically assesses generative NER using open-source LLMs across flat and nested NER tasks, focusing on output formats, memorization, and general capability preservation after instruction tuning. It finds that parameter-efficient fine-tuning with structured inline formats can match encoder-based NER and outperform GPT-3, with performance driven by instruction-following rather than memorization. The study also shows that instruction tuning largely preserves general multi-task abilities, and in some cases improves tasks requiring entity understanding like DROP. Overall, generative NER with LLMs emerges as a viable, user-friendly alternative to traditional methods, with broad practical impact for information extraction in diverse domains.

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass closed-source LLMs like GPT-3; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.
Paper Structure (16 sections, 3 figures, 13 tables)

This paper contains 16 sections, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instructions, Output Format, and Performance of Generative NER
  4. Settings
  5. Results
  6. Entity Recognition or Memorization
  7. Error Analysis
  8. Preservation of General Capability
  9. Conclusion
  10. Full Instructions for Different Datasets
  11. Instruction Construction Framework
  12. Task Descriptions for Different Output Formats
  13. Label Definition Schema
  14. Brief Introduction of Baselines
  15. Error Type Definitions
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Comparison between traditional NER based on pre-trained models (top) and generative NER based on large language models (bottom).
  • Figure 2: Distribution of error types for BERT-MRC and LLaMA3.1-8B on CoNLL2003.
  • Figure 3: The template and example of constructing instructions for generative NER.