UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition

TL;DR

This work introduces UniversalNER, a targeted distillation framework that trains compact models to excel at open-domain NER by distilling ChatGPT through mission-focused instruction tuning. It constructs a large, unlabeled data-driven instruction-tuning pipeline and evaluates on a diverse 43-dataset benchmark, achieving state-of-the-art performance without direct supervision and surpassing several instruction-tuned baselines. Supervised finetuning further boosts in-domain performance, while ablations highlight the importance of negative sampling and dataset-specific prompts. The results demonstrate robust cross-domain generalization and provide a practical distillation recipe and data release to advance targeted-Knowledge Distillation research.

Abstract

Large language models (LLMs) have demonstrated remarkable generalizability, such as understanding arbitrary entities and relations. Instruction tuning has proven effective for distilling LLMs into more cost-efficient models such as Alpaca and Vicuna. Yet such student models still trail the original LLMs by large margins in downstream applications. In this paper, we explore targeted distillation with mission-focused instruction tuning to train student models that can excel in a broad application class such as open information extraction. Using named entity recognition (NER) for case study, we show how ChatGPT can be distilled into much smaller UniversalNER models for open NER. For evaluation, we assemble the largest NER benchmark to date, comprising 43 datasets across 9 diverse domains such as biomedicine, programming, social media, law, finance. Without using any direct supervision, UniversalNER attains remarkable NER accuracy across tens of thousands of entity types, outperforming general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute F1 points in average. With a tiny fraction of parameters, UniversalNER not only acquires ChatGPT's capability in recognizing arbitrary entity types, but also outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably, UniversalNER even outperforms by a large margin state-of-the-art multi-task instruction-tuned systems such as InstructUIE, which uses supervised NER examples. We also conduct thorough ablation studies to assess the impact of various components in our distillation approach. We release the distillation recipe, data, and UniversalNER models to facilitate future research on targeted distillation.

Figures (9)

• Figure 1: Data construction prompt for generating entity mentions and their types for a given passage.
• Figure 2: The conversation-style template that converts a passage with NER annotations into a conversation, where ${\bm{X}}_\text{passage}$ is the input passage, $[{\bm{t}}_1,...,{\bm{t}}_T]$ are entity types to consider, and ${\bm{y}}_i$ is a list of entity mentions that are ${\bm{t}}_i$. The conversation is used to tune language models. Only the highlighted parts are used to compute the loss.
• Figure 3: The dataset-specific instruction tuning template. We add the dataset name ${\bm{D}}$ (colored in red) as part of the input to resolve conflicts in label definitions.
• Figure 4:
• Figure 5: Different in $F_1$ between data-specific and original templates in the supervised setting. Orange and Blue mark datasets with/without label overlap with other datasets, respectively.