Prompt-Learning for Fine-Grained Entity Typing

TL;DR

The paper addresses fine-grained entity typing by leveraging prompt-learning to bridge the gap between pre-training and fine-tuning for PLMs. It introduces a simple naive pipeline (Plet) that maps entity types to label words and uses declarative or learnable templates to cast typing as a masked language modeling task, with strong results in supervised and low-data regimes. For zero-shot scenarios, it adds a self-supervised prompt-learning variant that aligns prediction distributions over a restricted vocabulary using distribution-level optimization and unlabeled data, achieving notable gains. Across three challenging benchmarks (Few-NERD, OntoNotes, BBN), the approach demonstrates data-efficient improvements over vanilla fine-tuning and reveals important insights about template design and distribution-level learning for entity attribute extraction.

Abstract

As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{cloze}-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on three fine-grained entity typing benchmarks (with up to 86 classes) under fully supervised, few-shot and zero-shot settings show that prompt-learning methods significantly outperform fine-tuning baselines, especially when the training data is insufficient.

Figures (4)

• Figure 1: Examples of prompt-learning to stimulate the knowledge of PLMs by formalizing specific tasks as equivalent $cloze$-style tasks.
• Figure 2: The illustration of prompt-learning for fine-grained entity typing with supervision. We take hard-encoding prompt strategy as an example in this figure.
• Figure 3: The illustration of self-supervised prompt-learning for fine-grained entity typing with unlabeled data and a pre-defined label set. $\mathcal{V}^*$ denotes the label words projected from the input label set. Note that we only show the positive pair in this figure.
• Figure 4: Zero-shot prediction distribution on four types in Few-NERD, in each subgraph, the left part illustrates the results of Plet and the right part shows the results of Plet (S). denotes the correct predictions, denotes the wrong predictions with correct coarse-grained types, and denotes the wrong predictions with wrong coarse-grained types.