Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

Letian Peng, Zilong Wang, Feng Yao, Zihan Wang, Jingbo Shang

TL;DR

This work tackles the inefficiency of large LLMs for information extraction by distilling a universal meta-understanding of IE into a small model. MetaIE learns a label-to-span mapping $s=f_{IE}(X|l)$ through symbolic distillation from an LLM, using a diverse OpenWebText-based distillation dataset. Across 13 datasets and 6 IE tasks, MetaIE demonstrates strong few-shot transfer, outperforming single-task, multi-task human-annotated, and LLM-based baselines, with sequence-labeling backbones (notably RoBERTa) providing the best performance. The study highlights the importance of label distribution, data scale, and distillation framework in cross-task IE, suggesting practical, scalable benefits for rapid IE adaptation in real-world settings.

Abstract

Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.

MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

TL;DR

This work tackles the inefficiency of large LLMs for information extraction by distilling a universal meta-understanding of IE into a small model. MetaIE learns a label-to-span mapping through symbolic distillation from an LLM, using a diverse OpenWebText-based distillation dataset. Across 13 datasets and 6 IE tasks, MetaIE demonstrates strong few-shot transfer, outperforming single-task, multi-task human-annotated, and LLM-based baselines, with sequence-labeling backbones (notably RoBERTa) providing the best performance. The study highlights the importance of label distribution, data scale, and distillation framework in cross-task IE, suggesting practical, scalable benefits for rapid IE adaptation in real-world settings.

Abstract

Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.
Paper Structure (28 sections, 4 figures, 3 tables)

This paper contains 28 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Information Extraction
  4. Model Distillation
  5. Meta Learning
  6. Our MetaIE Framework
  7. Label-to-span Scheme
  8. Distillation Dataset Construction
  9. Implementation
  10. Distillation Framework
  11. Experiments
  12. IE Tasks and Datasets
  13. Evaluation Metric: Few-shot Fine-tuning Performance
  14. Compared Methods
  15. Result
  16. ...and 13 more sections

Figures (4)

  • Figure 1: An overview of different transfer learning schemes involved in the experiments.
  • Figure 2: The prompt used in our experiments to build the dataset for symbolic distillation.
  • Figure 3: The size analysis of the student model scale on different IE tasks and domains.
  • Figure 4: The size analysis of the distillation data scale on different IE tasks and domains.