Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NER

Andrew Zamai, Andrea Zugarini, Leonardo Rigutini, Marco Ernandes, Marco Maggini

TL;DR

This work proposes SLIMER, an approach designed to tackle never-seen-before entity tags by instructing the model on fewer examples, and by leveraging a prompt enriched with definition and guidelines, which performs comparably to state-of-the-art approaches in out-of-domain zero-shot NER.

Abstract

Recently, several specialized instruction-tuned Large Language Models (LLMs) for Named Entity Recognition (NER) have emerged. Compared to traditional NER approaches, these models have demonstrated strong generalization capabilities. Existing LLMs primarily focus on addressing zero-shot NER on Out-of-Domain inputs, while fine-tuning on an extensive number of entity classes that often highly or completely overlap with test sets. In this work instead, we propose SLIMER, an approach designed to tackle never-seen-before entity tags by instructing the model on fewer examples, and by leveraging a prompt enriched with definition and guidelines. Experiments demonstrate that definition and guidelines yield better performance, faster and more robust learning, particularly when labelling unseen named entities. Furthermore, SLIMER performs comparably to state-of-the-art approaches in out-of-domain zero-shot NER, while being trained in a more fair, though certainly more challenging, setting.

Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NER

TL;DR

This work proposes SLIMER, an approach designed to tackle never-seen-before entity tags by instructing the model on fewer examples, and by leveraging a prompt enriched with definition and guidelines, which performs comparably to state-of-the-art approaches in out-of-domain zero-shot NER.

Abstract

Recently, several specialized instruction-tuned Large Language Models (LLMs) for Named Entity Recognition (NER) have emerged. Compared to traditional NER approaches, these models have demonstrated strong generalization capabilities. Existing LLMs primarily focus on addressing zero-shot NER on Out-of-Domain inputs, while fine-tuning on an extensive number of entity classes that often highly or completely overlap with test sets. In this work instead, we propose SLIMER, an approach designed to tackle never-seen-before entity tags by instructing the model on fewer examples, and by leveraging a prompt enriched with definition and guidelines. Experiments demonstrate that definition and guidelines yield better performance, faster and more robust learning, particularly when labelling unseen named entities. Furthermore, SLIMER performs comparably to state-of-the-art approaches in out-of-domain zero-shot NER, while being trained in a more fair, though certainly more challenging, setting.
Paper Structure (37 sections, 5 figures, 8 tables)

This paper contains 37 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-Context Learning
  4. Fine-Tuning for Zero-Shot NER
  5. Definitions and guidelines for NER.
  6. SLIMER
  7. Show Less
  8. Instruct More
  9. Definition and Guidelines.
  10. Prompt structure.
  11. Output.
  12. Definition and Guidelines generation
  13. Experiments
  14. Datasets
  15. PileNER-type.
  16. ...and 22 more sections

Figures (5)

  • Figure 1: SLIMER's prompt. Dedicated entity definition and guidelines steer the model generation.
  • Figure 2: Prompt for generating the Definition and the Guidelines for a specific named entity.
  • Figure 3: Comparing SOTA models: $\mu$-F1 scores on unseen NEs in BUSTER (x-axis), OOD evaluation on MIT/CrossNER (y-axis). Circles' size is proportional to the number of examples seen in training by each model.
  • Figure 4: $\mu$ F1 scores of SLIMER and its baseline without D&G on MIT, CrossNER and BUSTER altogether, as we increase the number of unique NEs (top) and the number of samples per NE (bottom) seen in training.
  • Figure 5: The 391 Named Entities in the PileNER-type subset, grouped by macro topics. "misc" (not shown) groups 26 NEs that do not fit into the defined topics.