Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenderBench: Evaluation Suite for Gender Biases in LLMs

Matúš Pikuliak

TL;DR

GenderBench introduces a comprehensive, open-source evaluation suite to quantify gender biases in LLMs via 14 probes covering 19 harmful behaviors across outcomes, stereotypes, and representations. The authors provide a reusable library with bootstrapped metrics, prompt harnessing, and reporting, and benchmark 12 diverse LLMs, revealing convergent bias patterns and notable weaknesses in creative writing and stereotype-influenced reasoning while identifying cautious use in high-stakes decisions. The framework uses a four-tier severity scale to communicate results and emphasizes reproducibility, transparency, and extensibility, including plans to broaden coverage to languages, modalities, and non-binary genders. Overall, the work offers a practical, living benchmark to monitor and mitigate gender bias in LLMs and informs safer deployment in real-world applications.

Abstract

We present GenderBench -- a comprehensive evaluation suite designed to measure gender biases in LLMs. GenderBench includes 14 probes that quantify 19 gender-related harmful behaviors exhibited by LLMs. We release GenderBench as an open-source and extensible library to improve the reproducibility and robustness of benchmarking across the field. We also publish our evaluation of 12 LLMs. Our measurements reveal consistent patterns in their behavior. We show that LLMs struggle with stereotypical reasoning, equitable gender representation in generated texts, and occasionally also with discriminatory behavior in high-stakes scenarios, such as hiring.

GenderBench: Evaluation Suite for Gender Biases in LLMs

TL;DR

GenderBench introduces a comprehensive, open-source evaluation suite to quantify gender biases in LLMs via 14 probes covering 19 harmful behaviors across outcomes, stereotypes, and representations. The authors provide a reusable library with bootstrapped metrics, prompt harnessing, and reporting, and benchmark 12 diverse LLMs, revealing convergent bias patterns and notable weaknesses in creative writing and stereotype-influenced reasoning while identifying cautious use in high-stakes decisions. The framework uses a four-tier severity scale to communicate results and emphasizes reproducibility, transparency, and extensibility, including plans to broaden coverage to languages, modalities, and non-binary genders. Overall, the work offers a practical, living benchmark to monitor and mitigate gender bias in LLMs and informs safer deployment in real-world applications.

Abstract

We present GenderBench -- a comprehensive evaluation suite designed to measure gender biases in LLMs. GenderBench includes 14 probes that quantify 19 gender-related harmful behaviors exhibited by LLMs. We release GenderBench as an open-source and extensible library to improve the reproducibility and robustness of benchmarking across the field. We also publish our evaluation of 12 LLMs. Our measurements reveal consistent patterns in their behavior. We show that LLMs struggle with stereotypical reasoning, equitable gender representation in generated texts, and occasionally also with discriminatory behavior in high-stakes scenarios, such as hiring.
Paper Structure (45 sections, 3 figures, 4 tables)

This paper contains 45 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. GenderBench
  3. GenderBench Library
  4. GenderBench Benchmark
  5. Probes
  6. Bbq.
  7. BusinessVocabulary.
  8. Direct.
  9. DiscriminationTamkin.
  10. DiversityMedQa.
  11. Dreaddit.
  12. Gest.
  13. GestCreative.
  14. HiringAn.
  15. HiringBloomberg.
  16. ...and 30 more sections

Figures (3)

  • Figure 1: Detailed probe results for all the LLMs. The 95% confidence interval were calculated via bootstrapping. Colors are used to code the severity tiers: healthy , cautionary , critical , and catastrophic .
  • Figure 2: Pearson's correlation between LLMs based on normalized metrics.
  • Figure 3: Probe results for metrics that directly compare prefential treatment for women and men. The metrics always go from pro-female to pro-male with healthy values being in the middle.