Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Releasing Annotator-Level Labels and Information in Datasets

Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, Mark Díaz

TL;DR

The paper argues that aggregating multiple annotator judgments into a single ground truth can erase subjective perspectives and disproportionately bias datasets against minority viewpoints. It demonstrates this through analyses of eight tasks across three datasets, showing non-uniform annotator representation and demographic disparities in agreement with aggregated labels. The authors advocate releasing annotator-level labels and socio-demographic information (when feasible) and providing thorough documentation of recruitment and assignment to improve fairness and downstream research. These recommendations aim to enable more nuanced modeling of disagreement and to enhance transparency in subjective annotation tasks.

Abstract

A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single "ground truth" label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.

On Releasing Annotator-Level Labels and Information in Datasets

TL;DR

The paper argues that aggregating multiple annotator judgments into a single ground truth can erase subjective perspectives and disproportionately bias datasets against minority viewpoints. It demonstrates this through analyses of eight tasks across three datasets, showing non-uniform annotator representation and demographic disparities in agreement with aggregated labels. The authors advocate releasing annotator-level labels and socio-demographic information (when feasible) and providing thorough documentation of recruitment and assignment to improve fairness and downstream research. These recommendations aim to enable more nuanced modeling of disagreement and to enhance transparency in subjective annotation tasks.

Abstract

A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single "ground truth" label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.

Paper Structure

This paper contains 10 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Impacts of Aggregation
  4. Q1: Do Aggregated Labels Represent Individual Annotators Uniformly?
  5. Q2: Do Aggregated Labels Represent All Social Groups Uniformly?
  6. Utility of Annotator-level Labels
  7. Discussion and Conclusion
  8. Annotator-level labels:
  9. Socio-demographic information:
  10. Documentation about recruitment, selection, and assignment of annotators:

Figures (2)

  • Figure 1: Histograms represent the frequency distribution of annotator agreement with the aggregated label for eight tasks under three datasets for Emotions, Sentiment and Hate Speech datasets. The lack of uniformity in the distributions means that annotator perspectives are not equally captured in the majority labels.
  • Figure 2: Average and standard deviation of annotator agreement with aggregated labels, calculated for annotators grouped by their socio-demographics under gender, race, and political affiliation.