Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social Bias in Popular Question-Answering Benchmarks

Angelie Kraft, Judith Simon, Sonja Schimmler

TL;DR

The paper systematically analyzes 30 popular QA/RC benchmarks and 20 associated datasets to assess who creates benchmarks, what biases exist, and how biases relate to annotator demographics. It finds a pervasive lack of transparency, a Western-centric authorial and data representation bias, and multiple forms of demographic bias (gender, religion, location) across encyclopedic, commonsense, and scholarly benchmarks. The authors show that these biases can incentivize biased inference and epistemic injustice in downstream LLMs, challenging the fairness and generalizability of current evaluations. They advocate for explicit target distributions, transparent reporting, and active, non-exploitative involvement of marginalized communities in benchmark design. The work underscores the need for rigorous documentation and representational standards in AI evaluation to reduce biased optimization and improve equitable knowledge access.

Abstract

Question-answering (QA) and reading comprehension (RC) benchmarks are commonly used for assessing the capabilities of large language models (LLMs) to retrieve and reproduce knowledge. However, we demonstrate that popular QA and RC benchmarks do not cover questions about different demographics or regions in a representative way. We perform a content analysis of 30 benchmark papers and a quantitative analysis of 20 respective benchmark datasets to learn (1) who is involved in the benchmark creation, (2) whether the benchmarks exhibit social bias, or whether this is addressed or prevented, and (3) whether the demographics of the creators and annotators correspond to particular biases in the content. Most benchmark papers analyzed provide insufficient information about those involved in benchmark creation, particularly the annotators. Notably, just one (WinoGrande) explicitly reports measures taken to address social representation issues. Moreover, the data analysis revealed gender, religion, and geographic biases across a wide range of encyclopedic, commonsense, and scholarly benchmarks. Our work adds to the mounting criticism of AI evaluation practices and shines a light on biased benchmarks being a potential source of LLM bias by incentivizing biased inference heuristics.

Social Bias in Popular Question-Answering Benchmarks

TL;DR

The paper systematically analyzes 30 popular QA/RC benchmarks and 20 associated datasets to assess who creates benchmarks, what biases exist, and how biases relate to annotator demographics. It finds a pervasive lack of transparency, a Western-centric authorial and data representation bias, and multiple forms of demographic bias (gender, religion, location) across encyclopedic, commonsense, and scholarly benchmarks. The authors show that these biases can incentivize biased inference and epistemic injustice in downstream LLMs, challenging the fairness and generalizability of current evaluations. They advocate for explicit target distributions, transparent reporting, and active, non-exploitative involvement of marginalized communities in benchmark design. The work underscores the need for rigorous documentation and representational standards in AI evaluation to reduce biased optimization and improve equitable knowledge access.

Abstract

Question-answering (QA) and reading comprehension (RC) benchmarks are commonly used for assessing the capabilities of large language models (LLMs) to retrieve and reproduce knowledge. However, we demonstrate that popular QA and RC benchmarks do not cover questions about different demographics or regions in a representative way. We perform a content analysis of 30 benchmark papers and a quantitative analysis of 20 respective benchmark datasets to learn (1) who is involved in the benchmark creation, (2) whether the benchmarks exhibit social bias, or whether this is addressed or prevented, and (3) whether the demographics of the creators and annotators correspond to particular biases in the content. Most benchmark papers analyzed provide insufficient information about those involved in benchmark creation, particularly the annotators. Notably, just one (WinoGrande) explicitly reports measures taken to address social representation issues. Moreover, the data analysis revealed gender, religion, and geographic biases across a wide range of encyclopedic, commonsense, and scholarly benchmarks. Our work adds to the mounting criticism of AI evaluation practices and shines a light on biased benchmarks being a potential source of LLM bias by incentivizing biased inference heuristics.

Paper Structure

This paper contains 24 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Benchmark Selection
  5. Analysis of Benchmark Papers
  6. Analysis of Benchmark Datasets
  7. Results
  8. Benchmark Paper Analysis Results
  9. Benchmark Creation and Annotation
  10. Annotator Recruitment Criteria
  11. Annotator Demographics
  12. Benchmark Motivation
  13. Benchmark Bias and Toxicity
  14. Benchmark Language
  15. Benchmark Data Analysis Results
  16. ...and 9 more sections

Figures (7)

  • Figure 1: Qualitative content analysis process for the benchmark papers.
  • Figure 2: Quantitative data analysis process for the benchmark datasets.
  • Figure 3: Gender ratio for entities in encyclopedic, commonsense, and scholarly QA & RC benchmarks.
  • Figure 4: Distribution of domains across benchmarks.
  • Figure 5: Top-10 occupations by gender across benchmarks (if 300 or more occupations identified).
  • ...and 2 more figures