Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

Iñigo Parra

TL;DR

This work introduces UnMASKed, a linguistically informed framework to quantify gender biases in masked language models via two probing tasks: pronoun completion and linguistic unit prediction. By evaluating six MLMs (monolingual and multilingual) on a small, carefully designed dataset, the study reveals consistent gender stereotypes across models, with multilingual variants generally showing reduced bias. The combination of quantitative metrics (GTC, $\Delta$, $A$) and qualitative token-level analysis demonstrates that multilingual training can attenuate bias, while biases persist in several domains. The findings underscore the importance of diverse linguistic data and targeted bias mitigation strategies for developing fairer NLP systems.

Abstract

Language models (LMs) have become pivotal in the realm of technological advancements. While their capabilities are vast and transformative, they often include societal biases encoded in the human-produced datasets used for their training. This research delves into the inherent biases present in masked language models (MLMs), with a specific focus on gender biases. This study evaluated six prominent models: BERT, RoBERTa, DistilBERT, BERT-multilingual, XLM-RoBERTa, and DistilBERT-multilingual. The methodology employed a novel dataset, bifurcated into two subsets: one containing prompts that encouraged models to generate subject pronouns in English, and the other requiring models to return the probabilities of verbs, adverbs, and adjectives linked to the prompts' gender pronouns. The analysis reveals stereotypical gender alignment of all models, with multilingual variants showing comparatively reduced biases.

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

TL;DR

This work introduces UnMASKed, a linguistically informed framework to quantify gender biases in masked language models via two probing tasks: pronoun completion and linguistic unit prediction. By evaluating six MLMs (monolingual and multilingual) on a small, carefully designed dataset, the study reveals consistent gender stereotypes across models, with multilingual variants generally showing reduced bias. The combination of quantitative metrics (GTC, , ) and qualitative token-level analysis demonstrates that multilingual training can attenuate bias, while biases persist in several domains. The findings underscore the importance of diverse linguistic data and targeted bias mitigation strategies for developing fairer NLP systems.

Abstract

Language models (LMs) have become pivotal in the realm of technological advancements. While their capabilities are vast and transformative, they often include societal biases encoded in the human-produced datasets used for their training. This research delves into the inherent biases present in masked language models (MLMs), with a specific focus on gender biases. This study evaluated six prominent models: BERT, RoBERTa, DistilBERT, BERT-multilingual, XLM-RoBERTa, and DistilBERT-multilingual. The methodology employed a novel dataset, bifurcated into two subsets: one containing prompts that encouraged models to generate subject pronouns in English, and the other requiring models to return the probabilities of verbs, adverbs, and adjectives linked to the prompts' gender pronouns. The analysis reveals stereotypical gender alignment of all models, with multilingual variants showing comparatively reduced biases.
Paper Structure (18 sections, 2 equations, 3 figures, 3 tables)

This paper contains 18 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Methodology
  4. Datasets
  5. Model Selection
  6. Gender Bias Evaluation Criteria
  7. Quantitative Analysis
  8. Qualitative Analysis
  9. Results
  10. Gender Pronoun Completions
  11. Linguistic Token Completion
  12. BERT vs BERT-multilingual
  13. DistilBERT vs DistilBERT-multilingual
  14. RoBERTa vs XLM-RoBERTa
  15. Discussion and Future Work
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Summary of the approach. The sections in the diagram (from left to right) show the steps taken to judge each model.
  • Figure 2: Diagram of the dataset structure. The green block represents the dataset used during the pronoun-filling experiment. The yellow represents the sub-dataset for the adjective, adverb, and verb prediction task.
  • Figure 3: Total number of parallel pairs per model. The plot shows the number of token coincidences across linguistic units (adverbs, adjectives, and verbs) for male and female subject pronoun prompts.