Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Gender Bias in Turkish Language Models

Orhun Mersin Caglidil, Malte Ostendorff, Georg Rehm

TL;DR

This study probes gender and ethnic bias in Turkish language models by translating and extending established bias evaluation frameworks (WEAT/SEAT) to Turkish and by introducing Turkish-specific tests, including assessments of bias toward Kurdish. It evaluates several monolingual and multilingual Turkish LMs (e.g., BERTurk variants, mBERT, mT5) and analyzes how model size, multilingual pretraining, and training data influence bias, revealing that sentence-level tests generally uncover stronger biases and that multilingual models often exhibit different bias patterns than monolingual ones. The results show limited Kurdish-Turkish ethnic bias across most models, with mT5 showing some sensitivity potentially tied to its diverse training data; larger models tend to display stronger gender biases in several double-bind scenarios, and uncased models sometimes diverge from their cased counterparts. The authors provide a publicly released Turkish bias dataset and code, discuss methodological limitations (notably the translation-grounded nature of the tests and Turkish linguistic features), and propose future work to refine Turkish-specific bias measures and broaden gender representations. The work advances cross-linguistic bias understanding and offers resources for evaluating and mitigating bias in Turkish NLP systems, with implications for fairer downstream applications in Türkiye.

Abstract

Language models are trained mostly on Web data, which often contains social stereotypes and biases that the models can inherit. This has potentially negative consequences, as models can amplify these biases in downstream tasks or applications. However, prior research has primarily focused on the English language, especially in the context of gender bias. In particular, grammatically gender-neutral languages such as Turkish are underexplored despite representing different linguistic properties to language models with possibly different effects on biases. In this paper, we fill this research gap and investigate the significance of gender bias in Turkish language models. We build upon existing bias evaluation frameworks and extend them to the Turkish language by translating existing English tests and creating new ones designed to measure gender bias in the context of Türkiye. Specifically, we also evaluate Turkish language models for their embedded ethnic bias toward Kurdish people. Based on the experimental results, we attribute possible biases to different model characteristics such as the model size, their multilingualism, and the training corpora. We make the Turkish gender bias dataset publicly available.

Investigating Gender Bias in Turkish Language Models

TL;DR

This study probes gender and ethnic bias in Turkish language models by translating and extending established bias evaluation frameworks (WEAT/SEAT) to Turkish and by introducing Turkish-specific tests, including assessments of bias toward Kurdish. It evaluates several monolingual and multilingual Turkish LMs (e.g., BERTurk variants, mBERT, mT5) and analyzes how model size, multilingual pretraining, and training data influence bias, revealing that sentence-level tests generally uncover stronger biases and that multilingual models often exhibit different bias patterns than monolingual ones. The results show limited Kurdish-Turkish ethnic bias across most models, with mT5 showing some sensitivity potentially tied to its diverse training data; larger models tend to display stronger gender biases in several double-bind scenarios, and uncased models sometimes diverge from their cased counterparts. The authors provide a publicly released Turkish bias dataset and code, discuss methodological limitations (notably the translation-grounded nature of the tests and Turkish linguistic features), and propose future work to refine Turkish-specific bias measures and broaden gender representations. The work advances cross-linguistic bias understanding and offers resources for evaluating and mitigating bias in Turkish NLP systems, with implications for fairer downstream applications in Türkiye.

Abstract

Language models are trained mostly on Web data, which often contains social stereotypes and biases that the models can inherit. This has potentially negative consequences, as models can amplify these biases in downstream tasks or applications. However, prior research has primarily focused on the English language, especially in the context of gender bias. In particular, grammatically gender-neutral languages such as Turkish are underexplored despite representing different linguistic properties to language models with possibly different effects on biases. In this paper, we fill this research gap and investigate the significance of gender bias in Turkish language models. We build upon existing bias evaluation frameworks and extend them to the Turkish language by translating existing English tests and creating new ones designed to measure gender bias in the context of Türkiye. Specifically, we also evaluate Turkish language models for their embedded ethnic bias toward Kurdish people. Based on the experimental results, we attribute possible biases to different model characteristics such as the model size, their multilingualism, and the training corpora. We make the Turkish gender bias dataset publicly available.
Paper Structure (20 sections, 4 equations, 1 figure, 2 tables)

This paper contains 20 sections, 4 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. (1) What kind of gender stereotypes do the language models pick up? What are some common patterns for the errors that LMs make in this regard?
  3. (2) How can these patterns be explained by model properties?
  4. Related Work
  5. Methodology
  6. Evaluation Framework
  7. Word Embeddings Association Test.
  8. Sentence Encoder Association Test.
  9. Turkish Bias Tests
  10. Caliskan Tests.
  11. Double Bind Tests.
  12. Results
  13. Caliskan Tests
  14. Association of Kurdish/Turkish names with Pleasant/Unpleasant Attributes.
  15. Association between male/female and career/family/science/art attributes.
  16. ...and 5 more sections

Figures (1)

  • Figure 1: P-values for a selected relevant subset of Turkish gender bias tests including WEAT, SEAT, and Double Bind (DB) tests. The tests using the group-terms are annotated with a 'b' in the test name. WEAT tests illustrate that monolingual models elicit biased associations for given-name tests while the multilingual models demonstrate opposite behavior. Sentence-level SEAT4 test checking for the association of Turkish/Kurdish names with pleasant/unpleasent attributes, where high p-values indicate that the result is statistically insignificant. Double Bind tests show that the statistical significance increases when using traditional set of names instead of the mixed set of names.