Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

TL;DR

This work introduces Filipino CrowS-Pairs and Filipino WinoQueer to quantify sexist and homophobic bias in multilingual language models operating in Filipino. It documents a rigorous cultural adaptation pipeline from English benchmarks to Filipino, addressing linguistic gender, sexuality concepts, and cultural context. Evaluations across masked and causal multilingual PLMs reveal persistent bias, with higher bias linked to greater exposure to Filipino data in pretraining and elements influenced by English-derived terminology. The resulting benchmarks enable cross-language bias analysis and guide debiasing efforts for low-resource languages in Southeast Asia.

Abstract

Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.