What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models

TL;DR

This work tackles gender bias in multilingual masked language models by introducing a non-parallel, language-agnostic evaluation framework built on a multilingual gender lexicon (MGL) and two sentence-generation strategies (LSG, MSG). It defines three metrics (MBE, SBM, DBM) to measure bias while mitigating confounds, preserving data diversity, and enabling cross-language comparisons. The study reveals data sensitivity in prior English-centric methods and shows that SBM and DBM can yield different bias directions across languages and generation methods, with MSG offering greater stability and data efficiency. Overall, the framework provides a practical, multi-faceted approach for reliable bias assessment in multilingual NLP systems, with implications for fairer deployment and broader linguistic coverage; thresholds and parameters such as $k=10$ and $threshold=0.01$ guided model-based generation diversity and confidence filtering.

Abstract

Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (MLMs) and their impact on many NLP tasks, a systematic evaluation of bias in these models is needed more than ever. While many studies have evaluated gender bias in English MLMs, only a few works have been conducted for the task in other languages. This paper proposes a multilingual approach to estimate gender bias in MLMs from 5 languages: Chinese, English, German, Portuguese, and Spanish. Unlike previous work, our approach does not depend on parallel corpora coupled with English to detect gender bias in other languages using multilingual lexicons. Moreover, a novel model-based method is presented to generate sentence pairs for a more robust analysis of gender bias, compared to the traditional lexicon-based method. For each language, both the lexicon-based and model-based methods are applied to create two datasets respectively, which are used to evaluate gender bias in an MLM specifically trained for that language using one existing and 3 new scoring metrics. Our results show that the previous approach is data-sensitive and not stable as it does not remove contextual dependencies irrelevant to gender. In fact, the results often flip when different scoring metrics are used on the same dataset, suggesting that gender bias should be studied on a large dataset using multiple evaluation metrics for best practice.

Figures (4)

• Figure 1: An overview of the lexicon-based sentence extraction (Section \ref{['sec:rule']}).
• Figure 2: An overview of the model-based sentence extraction (Section \ref{['sec:model-based']}) using the examples in Figure \ref{['fig:lexicon-based']}.
• Figure 3: The proportions of sentences containing gender words whose confidence scores are higher than the threshold $0.01$ covered by the $k = [1, 15]$ predictions.
• Figure 4: The numbers of gender word types generated by LSG and MSG. MSG: One|Both: the number of gender word types where MSG generates for only one gender or both genders, respectively.