Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

Benedetta Muscato, Lucia Passaro, Gizem Gezici, Fosca Giannotti

TL;DR

This paper argues that aggregating annotator disagreement into a single ground truth obscures minority perspectives in subjective NLP tasks and proposes a multi-perspective, soft-label framework to preserve diverse annotations. By learning from disaggregated label distributions with a soft loss, the approach improves alignment with human label distributions (via Jensen-Shannon Divergence) and often enhances macro-F1 across hate speech, abusive language, irony, and stance detection, while acknowledging higher uncertainty in inherently subjective cases. The work formalizes the soft-label objective, evaluates on four datasets, and complements quantitative results with an Explainable AI analysis to study model uncertainty and token-level explanations. Overall, it advocates a pluralistic, perspective-aware direction for more inclusive and trustworthy NLP systems and outlines practical steps for broader adoption and future research.

Abstract

In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.

Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

TL;DR

This paper argues that aggregating annotator disagreement into a single ground truth obscures minority perspectives in subjective NLP tasks and proposes a multi-perspective, soft-label framework to preserve diverse annotations. By learning from disaggregated label distributions with a soft loss, the approach improves alignment with human label distributions (via Jensen-Shannon Divergence) and often enhances macro-F1 across hate speech, abusive language, irony, and stance detection, while acknowledging higher uncertainty in inherently subjective cases. The work formalizes the soft-label objective, evaluates on four datasets, and complements quantitative results with an Explainable AI analysis to study model uncertainty and token-level explanations. Overall, it advocates a pluralistic, perspective-aware direction for more inclusive and trustworthy NLP systems and outlines practical steps for broader adoption and future research.

Abstract

In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.

Paper Structure

This paper contains 22 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Perspective-aware Datasets
  4. Perspective-aware Models
  5. Methodology
  6. Baseline Models using Hard Labels
  7. Majority Vote
  8. Ensemble
  9. Multi-Perspective Model using Soft Labels
  10. Experimental Setup
  11. Datasets
  12. GabHate
  13. ConvAbuse
  14. EPIC
  15. StanceDetection
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Overview of baselines, left: majority vote, middle: ensemble and right: the multi-perspective approach, which uses soft labels shown as color gradients. Annotator counts are illustrative; actual benchmarks range from 3 to 18 annotators.
  • Figure 2: Three XAI methods applied to a low-confidence instance identified by the best multi-perspective model on ConvAbuse.
  • Figure 3: Three XAI methods applied to a low-confidence instance identified by the best baseline model on EPIC.