Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

TL;DR

The paper addresses annotator disagreement in NLP, arguing that single ground-truth labels obscure legitimate diverse interpretations. It proposes a perspectivist framework with a taxonomy of data-, task-, and annotator-driven sources and reviews three modeling families for disagreement: latent truth with reliability, task-based multi-annotator models, and embedding-based annotator models. Evaluation is discussed through probabilistic and fairness-oriented metrics, noting a trend toward modeling the full disagreement distribution and subgroup perspectives rather than only predicting a consensus label. Open challenges include integrating multiple sources of variation, improving interpretability, and developing normative guidelines for which perspectives to preserve in real-world NLP systems.

Abstract

Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.

Figures (4)

• Figure 1: Taxonomy of sources of annotator disagreement, with key citations for each sub-category. For each source of disagreement, we denote systematic variation using solid lines, and random variation using dotted.
• Figure 2: The distribution of the three main methods across our identified taxonomy of sources of disagreement. The lack of work modeling data and task factors point to future directions.
• Figure 3: Prediction targets and pooling assumptions across methods for learning from annotator disagreement. The table maps existing approaches by what they predict (consensus labels, individual annotator responses, group-level outputs, or full disagreement distributions) and how they pool information across annotators. Fig. \ref{['fig:syn_detailed']} is a more detailed table with representative work.
• Figure 4: Prediction targets and pooling assumptions across methods for learning from annotator disagreement. The table maps existing approaches by what they predict (consensus labels, individual annotator responses, group-level outputs, or full disagreement distributions) and how they pool information across annotators. Modeling choices implicitly encode different assumptions about population structure, perspective aggregation with recent work increasingly favoring partial pooling and distributional targets.