MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

TL;DR

The paper investigates whether social biases in generative LLMs depend on the language used to prompt, beyond cultural differences. It introduces MBBQ, a hand-checked Dutch, Spanish, and Turkish translation of BBQ, plus a parallel control set to separate task performance from bias, and evaluates seven chat-optimized LLMs using five prompts per language. The study reports significant cross-lingual variation in accuracy and bias, with biases often strongest in Spanish and generally reduced for the most accurate models in English or Turkish; biases also vary by stereotype category. The work provides a robust multilingual bias benchmark and methodology, enabling cross-lingual debiasing research and careful interpretation of bias metrics alongside task performance. Overall, MBBQ offers a valuable framework for assessing and mitigating stereotypes in multilingual LLM usage and highlights the need for language-aware fairness evaluation.

Abstract

Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at https://github.com/Veranep/MBBQ.

Figures (4)

• Figure 1: Example from the Gender identity category in English and Spanish, plus the control-MBBQ counterpart of the English sample. In ambiguous contexts, the correct answer is always the "unknown" answer. In biased contexts, the correct answer to this question is the biased answer, and in counter-biased contexts this is the counter-biased answer.
• Figure 2: Percentage of templates in control-MBBQ where models perform above chance (33%) and accuracy on those templates. Models sorted by their accuracy in English.
• Figure 3: Bias scores in ambiguous contexts per subset. Bias scores that are significantly different from 0 ($p < 0.05$) are marked with a star ($\star$).
• Figure 4: Bias scores in disambiguated contexts per subset. Bias scores that are significantly different from 0 ($p < 0.05$) are marked with a star (*).