Out of Sight Out of Mind, Out of Sight Out of Mind: Measuring Bias in Language Models Against Overlooked Marginalized Groups in Regional Contexts

TL;DR

The paper investigates offensive stereotyping bias (SOS) against overlooked marginalized groups across 25 countries and three languages, using 23 LMs and datasets that cover 270 marginalized groups. It introduces a bilingual Arabic data framework (MSA and Egyptian), English, and German, plus a new SOS bias metric for MLMs and extended evaluation on IFMs and Generative Models, highlighting dialect- and region-specific biases. Key findings include pronounced SOS bias for Egyptian Arabic and substantial intersectional bias (e.g., LGBTQIA+ and Black women), with Arabic MLMs showing high bias for religion and ethnicity, while English/German LMs exhibit stronger biases for intersectional identities. The study also reveals limitations in current bias metrics for low-resource languages and emphasizes the need for locally sourced data to train truly inclusive multilingual LMs, as translation-based data can propagate existing stereotypes.

Abstract

We know that language models (LMs) form biases and stereotypes of minorities, leading to unfair treatments of members of these groups, thanks to research mainly in the US and the broader English-speaking world. As the negative behavior of these models has severe consequences for society and individuals, industry and academia are actively developing methods to reduce the bias in LMs. However, there are many under-represented groups and languages that have been overlooked so far. This includes marginalized groups that are specific to individual countries and regions in the English speaking and Western world, but crucially also almost all marginalized groups in the rest of the world. The UN estimates, that between 600 million to 1.2 billion people worldwide are members of marginalized groups and in need for special protection. If we want to develop inclusive LMs that work for everyone, we have to broaden our understanding to include overlooked marginalized groups and low-resource languages and dialects. In this work, we contribute to this effort with the first study investigating offensive stereotyping bias in 23 LMs for 270 marginalized groups from Egypt, the remaining 21 Arab countries, Germany, the UK, and the US. Additionally, we investigate the impact of low-resource languages and dialects on the study of bias in LMs, demonstrating the limitations of current bias metrics, as we measure significantly higher bias when using the Egyptian Arabic dialect versus Modern Standard Arabic. Our results show, LMs indeed show higher bias against many marginalized groups in comparison to dominant groups. However, this is not the case for Arabic LMs, where the bias is high against both marginalized and dominant groups in relation to religion and ethnicity. Our results also show higher intersectional bias against Non-binary, LGBTQIA+ and Black women.

Figures (9)

• Figure 1: SOS bias scores (using the HONEST metric) in generative models
• Figure 2: The distribution of bias scores in MLMs against identities in the Arab world. The full results for all regions are in Appendix \ref{['appx:detailed_results_MLM']}
• Figure 3: Heatmap of the SOS bias scores against the refugees/nationals (Male) in Germany and Egypt.
• Figure 4: SOS bias scores in the BART model for Egypt and the UK for all genders.
• Figure 5: HONEST scores in all generative models for the UK and all genders.