CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition

TL;DR

CLLMFS addresses few-shot NER by integrating LoRA-based supervised fine-tuning of a large language model with a tailored contrastive learning objective and adversarial embedding augmentation to enhance entity boundary awareness and recognition accuracy. It leverages carefully designed SFT data and a constrained decoding strategy to align model outputs with input-derived entity spans, while using InfoNCE-based contrastive losses computed at a mid-to-deep layer to refine internal representations. Across five public FS-NER datasets and cross-domain evaluations, CLLMFS achieves state-of-the-art F1 gains and demonstrates robust transfer capabilities in low-resource settings. The approach offers practical benefits for real-world NER in diverse domains and lays groundwork for extending to relation extraction and broader information extraction tasks.

Abstract

Few-shot Named Entity Recognition (NER), the task of identifying named entities with only a limited amount of labeled data, has gained increasing significance in natural language processing. While existing methodologies have shown some effectiveness, such as enriching label semantics through various prompting modes or employing metric learning techniques, their performance exhibits limited robustness across diverse domains due to the lack of rich knowledge in their pre-trained models. To address this issue, we propose CLLMFS, a Contrastive Learning enhanced Large Language Model (LLM) Framework for Few-Shot Named Entity Recognition, achieving promising results with limited training data. Considering the impact of LLM's internal representations on downstream tasks, CLLMFS integrates Low-Rank Adaptation (LoRA) and contrastive learning mechanisms specifically tailored for few-shot NER. By enhancing the model's internal representations, CLLMFS effectively improves both entity boundary awareness ability and entity recognition accuracy. Our method has achieved state-of-the-art performance improvements on F1-score ranging from 2.58\% to 97.74\% over existing best-performing methods across several recognized benchmarks. Furthermore, through cross-domain NER experiments conducted on multiple datasets, we have further validated the robust generalization capability of our method. Our code will be released in the near future.

Figures (1)

• Figure 1: The framework overview CLLMFS. The LLM extracts named entities from carefully designed SFT data using decoding strategies, LoRA fine-tuning leveraging LLM's attention mechanisms such as QKV computations, constructing positive and negative samples for contrastive learning, and creating adversarial embedding samples.