Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training

TL;DR

Few-shot relation extraction struggles with limited labeled data and weak generalization. The paper introduces TKRE, a Two-Stage Knowledge-Guided framework that bridges generative and discriminative learning by using LLMs to produce explanation-driven knowledge and schema-constrained synthetic data, followed by a two-stage pre-training that combines Masked Span Language Modeling ($L_{\mathrm{mslm}}$) and Span-Level Contrastive Learning ($L_{\mathrm{scl}}$). This is followed by task-oriented fine-tuning on both golden and synthetic data. Across four benchmark datasets, TKRE achieves state-of-the-art FSRE performance, with ablations confirming the contributions of explanation-driven data, schema constraints, and span-level pre-training. The approach demonstrates strong practical potential for low-resource relation extraction and provides a blueprint for integrating generative knowledge into discriminative IR pipelines, scalable to additional domains and tasks.

Abstract

Few-Shot Relation Extraction (FSRE) remains a challenging task due to the scarcity of annotated data and the limited generalization capabilities of existing models. Although large language models (LLMs) have demonstrated potential in FSRE through in-context learning (ICL), their general-purpose training objectives often result in suboptimal performance for task-specific relation extraction. To overcome these challenges, we propose TKRE (Two-Stage Knowledge-Guided Pre-training for Relation Extraction), a novel framework that synergistically integrates LLMs with traditional relation extraction models, bridging generative and discriminative learning paradigms. TKRE introduces two key innovations: (1) leveraging LLMs to generate explanation-driven knowledge and schema-constrained synthetic data, addressing the issue of data scarcity; and (2) a two-stage pre-training strategy combining Masked Span Language Modeling (MSLM) and Span-Level Contrastive Learning (SCL) to enhance relational reasoning and generalization. Together, these components enable TKRE to effectively tackle FSRE tasks. Comprehensive experiments on benchmark datasets demonstrate the efficacy of TKRE, achieving new state-of-the-art performance in FSRE and underscoring its potential for broader application in low-resource scenarios. \footnote{The code and data are released on https://github.com/UESTC-GQJ/TKRE.

Figures (6)

• Figure 1: (a) Distant supervision is an automated and efficient approach for generating labeled data, but its effectiveness depends on the quality of the knowledge base. Challenges such as noisy labels and context loss are common issues with distant supervision. (b) In contrast, our approach leverages explanation-driven knowledge and synthetic data automatically generated by LLMs, which significantly enhance the performance of small language models (SLMs).
• Figure 2: Overall framework of the proposed TKRE. (a) Leveraging LLMs to generate explanation-driven knowledge and schema-constrained synthetic data. (b) Implementing a two-stage knowledge-guided pre-training framework to enhance the model’s ability to understand relational structures and contextual dependencies. (c) Performing task-oriented fine-tuning and evaluation on the test dataset to validate the model’s effectiveness.
• Figure 3: Instruction of explanation-driven knowledge generation.
• Figure 4: Instruction of schema-constrained synthetic data generation.
• Figure 5: Micro F1 (%) of TKRE (GenPT) with generated explanation-driven knowledge by various LLMs.