Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction

TL;DR

This work explores the definition only zero-shot RE setting where only relation definitions expressed in natural language are used to train a RE model, and introduces REPaL, which significantly improved cost-effective zero-shot performance by large margins.

Abstract

Relation extraction (RE) aims to identify semantic relationships between entities within text. Despite considerable advancements, existing models predominantly require extensive annotated training data, which is both costly and labor-intensive to collect. Moreover, these models often struggle to adapt to new or unseen relations. Few-shot learning, aiming to lessen annotation demands, typically provides incomplete and biased supervision for target relations, leading to degraded and unstable performance. To accurately and explicitly describe relation semantics while minimizing annotation demands, we explore the definition only zero-shot RE setting where only relation definitions expressed in natural language are used to train a RE model. We introduce REPaL, comprising three stages: (1) We leverage large language models (LLMs) to generate initial seed instances from relation definitions and an unlabeled corpus. (2) We fine-tune a bidirectional Small Language Model (SLM) with initial seeds to learn relations for the target domain. (3) We expand pattern coverage and mitigate bias from initial seeds by integrating feedback from the SLM's predictions on the unlabeled corpus and the synthesis history. To accomplish this, we leverage the multi-turn conversation ability of LLMs to generate new instances in follow-up dialogues, informed by both the feedback and synthesis history. Studies reveal that definition-oriented seed synthesis enhances pattern coverage whereas indiscriminately increasing seed quantity leads to performance saturation. Experiments on two datasets show REPaL significantly improved cost-effective zero-shot performance by large margins.

Figures (9)

• Figure 1: Different types of initial seeds for low-resource RE approaches for example relation P276. It shows using only two instances as seeds fail to cover structure type head entities.
• Figure 2: Micro $F_1$ (%) score of model trained with few-shot instances (Trained w Few-Shot Ex) and model trained with instances from our relation definition derivation and instance generation approach (Def Deduce+Ex Gen). The error bar/band denotes averaged value $\pm$ standard deviation.
• Figure 3: $F_1$ (%) scores for different setups on the number and ratio of training instances.
• Figure 4: REPaL framework. The trained SLM-Based RE Model is used in inference stage.
• Figure 5: Precision (%) scores for different setups on the number and ratio of training instances.