Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation

TL;DR

This paper interrogates the data annotation pipeline in ML, arguing that 'ground truth' is a sociotechnical artifact produced through procedural norms that erase disagreement and lived experience. It uses a PRISMA-based systematic review across seven venues (2020–2025) to synthesize 346 papers, revealing mechanisms like annotator positionality, labor precarity, Western geographic hegemony, and model-as-annotator dynamics that foster a consensus trap. The authors advocate reclaiming disagreement as a high-fidelity signal and propose a roadmap for pluralistic annotation infrastructures featuring perspectivist adjudication, rationale extraction, and community-centric participatory design to realize epistemic justice. The work highlights cross-disciplinary tensions and calls for infrastructural and organizational reforms to shift from extractive data labor toward situated knowledge stewardship, with practical guidance for researchers, practitioners, and platforms to build culturally competent AI systems.

Abstract

In machine learning, "ground truth" refers to the assumed correct labels used to train and evaluate models. However, the foundational "ground truth" paradigm rests on a positivistic fallacy that treats human disagreement as technical noise rather than a vital sociotechnical signal. This systematic literature review analyzes research published between 2020 and 2025 across seven premier venues: ACL, AIES, CHI, CSCW, EAAMO, FAccT, and NeurIPS, investigating the mechanisms in data annotation practices that facilitate this "consensus trap". Our identification phase captured 30,897 records, which were refined via a tiered keyword filtration schema to a high-recall corpus of 3,042 records for manual screening, resulting in a final included corpus of 346 papers for qualitative synthesis. Our reflexive thematic analysis reveals that systemic failures in positional legibility, combined with the recent architectural shift toward human-as-verifier models, specifically the reliance on model-mediated annotations, introduce deep-seated anchoring bias and effectively remove human voices from the loop. We further demonstrate how geographic hegemony imposes Western norms as universal benchmarks, often enforced by the performative alignment of precarious data workers who prioritize requester compliance over honest subjectivity to avoid economic penalties. Critiquing the "noisy sensor" fallacy, where statistical models misdiagnose cultural pluralism as random error, we argue for reclaiming disagreement as a high-fidelity signal essential for building culturally competent models. To address these systemic tensions, we propose a roadmap for pluralistic annotation infrastructures that shift the objective from discovering a singular "right" answer to mapping the diversity of human experience.

Figures (2)

• Figure 1: Overview of the review workflow. We first formulate research questions and define a PICOC framework with inclusion/exclusion criteria, then identify records that are passed through keyword filtration, title/abstract screening, and full-text review, leading to thematic analysis and synthesis of the final corpus (346 papers).
• Figure 2: PRISMA Flow Diagram of the Systematic Review, showing record identification, keyword filtration, and screening stages leading to the final corpus of 346 papers 19. Full-text exclusion reasons are reported in Appendix \ref{['app:exclusions']}.