Multi-Perspective Stance Detection

TL;DR

This work tackles subjectivity in stance detection by comparing traditional single-label training with a multi-perspective approach that preserves annotator diversity. It formalizes a Baseline using a single majority label $m_i$ and a Multi-perspective setup using per-annotator labels $A(d_i)=\{a_1,a_2,a_3\}$ to generate multiple training instances ${d_i}^k=\{q_i,c_i,a_k\}$, coupled with chunking to handle long documents. Evaluations on a dataset of 1062 cleaned items (labels: pro, neutral, against, not-about, link not-working) show that multi-perspective models generally outperform baselines and that chunking enhances both performance and confidence across transformer-based encoders (BERT-base, RoBERTa-base). The findings advocate for perspectivist data collection and disaggregated labels as a means to improve both fairness and accuracy in subjective NLP tasks, with future work extending to larger models, tuned augmentation strategies, and related tasks like ideology detection.

Abstract

Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.

Figures (2)

• Figure 1: Baseline vs. Multi-perspective Approach in Model Finetuning. Baseline relies on aggregated label via majority voting (i.e. Majority label), while Multi-perspective uses each annotator's individual label (i.e. Annotator label).
• Figure :