Relative Counterfactual Contrastive Learning for Mitigating Pretrained Stance Bias in Stance Detection

TL;DR

This work tackles pretrained stance bias in stance detection by introducing Relative Counterfactual Contrastive Learning (RCCL), which leverages a structural causal model, relative stance sample generation, and counterfactual contrastive learning to emphasize context-driven stance while suppressing bias from pretrained knowledge. The approach combines a do-calculus-inspired intervention with a margin-based contrastive objective, yielding state-of-the-art results on SemEval-2016, UKP, and VAST, and demonstrating robustness in few-shot and zero-shot scenarios. Ablation studies show that both the relative stance sampling (RSSG) and counterfactual contrastive learning (CCL) contribute substantially to performance, validating the proposed causal-debiasing strategy. Overall, RCCL offers a principled bias-mitigation framework for PLM-based NLP tasks by focusing on relative, counterfactual context rather than absolute pretrained features, with potential for broader applicability across languages and biases.

Abstract

Stance detection classifies stance relations (namely, Favor, Against, or Neither) between comments and targets. Pretrained language models (PLMs) are widely used to mine the stance relation to improve the performance of stance detection through pretrained knowledge. However, PLMs also embed ``bad'' pretrained knowledge concerning stance into the extracted stance relation semantics, resulting in pretrained stance bias. It is not trivial to measure pretrained stance bias due to its weak quantifiability. In this paper, we propose Relative Counterfactual Contrastive Learning (RCCL), in which pretrained stance bias is mitigated as relative stance bias instead of absolute stance bias to overtake the difficulty of measuring bias. Firstly, we present a new structural causal model for characterizing complicated relationships among context, PLMs and stance relations to locate pretrained stance bias. Then, based on masked language model prediction, we present a target-aware relative stance sample generation method for obtaining relative bias. Finally, we use contrastive learning based on counterfactual theory to mitigate pretrained stance bias and preserve context stance relation. Experiments show that the proposed method is superior to stance detection and debiasing baselines.

Figures (5)

• Figure 1: The examples of stance detection datasets. The stance distribution of BERT and GPT-2 for the same target "Feminist Movement" is opposite, which reveals pretrained stance bias.
• Figure 2: (a) Causal graph for stance detection. (b) Interventional stance detection where we directly model $P(Y|do(X=\hat{x}))$.
• Figure 3: The overall architecture of relative counterfactual contrastive learning.
• Figure 4: (a) The macro-f1 with data augmentation. (b) The macro-f1 with contrastive learning.
• Figure 5: The relationship between macro-F1 and masked ratio across different datasets. While the masked ratio is too high, we will fill in a part of random blanks at a time, iteratively until all masks are filled.