Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias

Enzo Doyen, Amalia Todirascu

TL;DR

Investigates masculine generics (MG) bias in LLM responses to generic French instructions, addressing a gap where prior work focused on constrained tasks. Builds a French Human Noun database using HScorer and analyzes MG usage across six LLMs with instruction filtering and M Score metrics. Finds MG appears in $39.5\%$ of responses overall and in $73.1\%$ of responses containing human nouns, with local models showing somewhat lower bias and a notable reluctance to use inclusive language. Demonstrates a replicable methodology and dataset that can be extended to other gendered languages, highlighting the need for bias mitigation and inclusive writing in LLM training and deployment.

Abstract

Large language models (LLMs) have been shown to propagate and even amplify gender bias, in English and other languages, in specific or constrained contexts. However, no studies so far have focused on gender biases conveyed by LLMs' responses to generic instructions, especially with regard to masculine generics (MG). MG are a linguistic feature found in many gender-marked languages, denoting the use of the masculine gender as a "default" or supposedly neutral gender to refer to mixed group of men and women, or of a person whose gender is irrelevant or unknown. Numerous psycholinguistics studies have shown that MG are not neutral and induce gender bias. This work aims to analyze the use of MG by both proprietary and local LLMs in responses to generic instructions and evaluate their MG bias rate. We focus on French and create a human noun database from existing lexical resources. We filter existing French instruction datasets to retrieve generic instructions and analyze the responses of 6 different LLMs. Overall, we find that $\approx$39.5\% of LLMs' responses to generic instructions are MG-biased ($\approx$73.1\% across responses with human nouns). Our findings also reveal that LLMs are reluctant to using gender-fair language spontaneously.

Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias

TL;DR

Investigates masculine generics (MG) bias in LLM responses to generic French instructions, addressing a gap where prior work focused on constrained tasks. Builds a French Human Noun database using HScorer and analyzes MG usage across six LLMs with instruction filtering and M Score metrics. Finds MG appears in of responses overall and in of responses containing human nouns, with local models showing somewhat lower bias and a notable reluctance to use inclusive language. Demonstrates a replicable methodology and dataset that can be extended to other gendered languages, highlighting the need for bias mitigation and inclusive writing in LLM training and deployment.

Abstract

Large language models (LLMs) have been shown to propagate and even amplify gender bias, in English and other languages, in specific or constrained contexts. However, no studies so far have focused on gender biases conveyed by LLMs' responses to generic instructions, especially with regard to masculine generics (MG). MG are a linguistic feature found in many gender-marked languages, denoting the use of the masculine gender as a "default" or supposedly neutral gender to refer to mixed group of men and women, or of a person whose gender is irrelevant or unknown. Numerous psycholinguistics studies have shown that MG are not neutral and induce gender bias. This work aims to analyze the use of MG by both proprietary and local LLMs in responses to generic instructions and evaluate their MG bias rate. We focus on French and create a human noun database from existing lexical resources. We filter existing French instruction datasets to retrieve generic instructions and analyze the responses of 6 different LLMs. Overall, we find that 39.5\% of LLMs' responses to generic instructions are MG-biased (73.1\% across responses with human nouns). Our findings also reveal that LLMs are reluctant to using gender-fair language spontaneously.

Paper Structure

This paper contains 22 sections, 10 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Database of French Human Nouns
  5. HScorer: a French Human Noun Classifier
  6. Analyzing MG Use in LLM Instructions and Outputs
  7. Instruction and Output Filtering
  8. LLM Instructing and Answer Retrieval
  9. Measuring MG Bias
  10. Results
  11. Discussion
  12. Conclusion
  13. Limitations
  14. Ethics Statement
  15. HScorer Scoring Functions
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Overview of our methodology. We filter human and AI-written instructions to remove specific contexts that could lead to masculine specific uses. We send these instructions to LLMs and retrieve their responses. LLM and human-written answers are analyzed for human-related MG (red) and inclusive or neutral (blue) terms. GPT-4o mini is used for human noun validation, and we use the final analysis results to calculate a M Score for each text.
  • Figure 2: Percentage of MG use rate by model
  • Figure 3: Overall (blue) and mean (red) M Score results for human-written and LLM responses by dataset/model
  • Figure 4: Frequency of unique MG nouns in responses based on their associated human noun class
  • Figure 5: Percentage of responses with inclusive language markers across models