Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning

TL;DR

Citss introduces a self-supervised contrastive learning framework to adapt pretrained language models for citation classification under limited labeled data. It employs two task-tailored transformations—sentence-level cropping and keyphrase perturbation—to generate positive pairs and mitigate noise and spurious keyphrase reliance, while being compatible with both encoder-based PLMs and decoder-based LLMs using PEFT. The approach achieves state-of-the-art or competitive results across three benchmark datasets, with strong ablations confirming the benefits of both SC and KP. The work advances practical citation analysis by enabling robust, scalable fine-tuning of diverse backbone models and providing a principled path to leverage large language models for scholarly text classification.

Abstract

Citation classification, which identifies the intention behind academic citations, is pivotal for scholarly analysis. Previous works suggest fine-tuning pretrained language models (PLMs) on citation classification datasets, reaping the reward of the linguistic knowledge they gained during pretraining. However, directly fine-tuning for citation classification is challenging due to labeled data scarcity, contextual noise, and spurious keyphrase correlations. In this paper, we present a novel framework, Citss, that adapts the PLMs to overcome these challenges. Citss introduces self-supervised contrastive learning to alleviate data scarcity, and is equipped with two specialized strategies to obtain the contrastive pairs: sentence-level cropping, which enhances focus on target citations within long contexts, and keyphrase perturbation, which mitigates reliance on specific keyphrases. Compared with previous works that are only designed for encoder-based PLMs, Citss is carefully developed to be compatible with both encoder-based PLMs and decoder-based LLMs, to embrace the benefits of enlarged pretraining. Experiments with three benchmark datasets with both encoder-based PLMs and decoder-based LLMs demonstrate our superiority compared to the previous state of the art. Our code is available at: github.com/LITONG99/Citss

Figures (6)

• Figure 1: Overview of Citss: (a) exhibits the architecture and workflow. (b) shows an example of sentence-level cropping. (c) shows an example of the keyphrase perturbation. The underlined text is the keyphrases and the shaded text is the word for synonym replacement.
• Figure 2: Ablation study of SC with different $T_i$. The x-axis is $l$, and $l=0$ corresponds to the citance.
• Figure 3: Performance with varying $\beta$ (x-axis) and $\gamma$. The dashed line is the performance without KP.
• Figure 4: Performance with perturbation operation $\mathsf{Op}$.
• Figure 5: Extracted STKs estimated by humans.