Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors

TL;DR

The paper tackles CFRE, where models must continually acquire new relations from limited data while preserving old knowledge. It introduces Contrastive Prompt Learning (CPL), combining a semi-automated prompt template with a margin-based contrastive loss to cultivate generalized representations, plus a memory augmentation pipeline that leverages LLM-generated samples to combat overfitting in low-resource regimes. The approach uses a memory-based rehearsal strategy and a Nearest-Class-Mean classifier at inference, showing strong empirical gains on FewRel and TACRED compared to state-of-the-art baselines. Overall, CPL demonstrates that prompt-based representations, coupled with targeted contrastive objectives and data augmentation via LLMs, can significantly mitigate forgetting and overfitting in continual few-shot relation extraction, with practical implications for scalable NLP systems.

Abstract

Continual Few-shot Relation Extraction (CFRE) is a practical problem that requires the model to continuously learn novel relations while avoiding forgetting old ones with few labeled training data. The primary challenges are catastrophic forgetting and overfitting. This paper harnesses prompt learning to explore the implicit capabilities of pre-trained language models to address the above two challenges, thereby making language models better continual few-shot relation extractors. Specifically, we propose a Contrastive Prompt Learning framework, which designs prompt representation to acquire more generalized knowledge that can be easily adapted to old and new categories, and margin-based contrastive learning to focus more on hard samples, therefore alleviating catastrophic forgetting and overfitting issues. To further remedy overfitting in low-resource scenarios, we introduce an effective memory augmentation strategy that employs well-crafted prompts to guide ChatGPT in generating diverse samples. Extensive experiments demonstrate that our method outperforms state-of-the-art methods by a large margin and significantly mitigates catastrophic forgetting and overfitting in low-resource scenarios.

Figures (6)

• Figure 1: Overview of catastrophic forgetting and overfitting in CFRE.
• Figure 2: Framework of the proposed CPL.
• Figure 3: Results of different prompt formats.
• Figure 4: t-SNE plots of instance embeddings trained with SCL and MCL.
• Figure 5: Results with different generated samples.