Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages

Viktor Mihaylov, Aleksandar Shtedritski

TL;DR

The paper investigates whether grammatical gender biases observed in human cognition generalize to multilingual LLMs. It prompts LLMs to describe gendered nouns with adjectives across ten languages and trains a binary classifier to predict noun gender from adjective descriptions. The study finds that the classifier's predictions transfer across languages, indicating similar biases despite language-specific adjective choices. These findings highlight cross-language biases in multilingual LLMs that could affect translation and anthropomorphism, underscoring the value of cross-linguistic psycholinguistic probes for AI systems.

Abstract

This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. Drawing inspiration from seminal works in psycholinguistics, particularly the study of gender's influence on language perception, we leverage multilingual LLMs to revisit and expand upon the foundational experiments of Boroditsky (2003). Employing LLMs as a novel method for examining psycholinguistic biases related to grammatical gender, we prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. In particular, we look at adjective co-occurrences across gender and languages, and train a binary classifier to predict grammatical gender given adjectives an LLM uses to describe a noun. Surprisingly, we find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability. We show that while LLMs may describe words differently in different languages, they are biased similarly.

What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages

TL;DR

The paper investigates whether grammatical gender biases observed in human cognition generalize to multilingual LLMs. It prompts LLMs to describe gendered nouns with adjectives across ten languages and trains a binary classifier to predict noun gender from adjective descriptions. The study finds that the classifier's predictions transfer across languages, indicating similar biases despite language-specific adjective choices. These findings highlight cross-language biases in multilingual LLMs that could affect translation and anthropomorphism, underscoring the value of cross-linguistic psycholinguistic probes for AI systems.

Abstract

This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. Drawing inspiration from seminal works in psycholinguistics, particularly the study of gender's influence on language perception, we leverage multilingual LLMs to revisit and expand upon the foundational experiments of Boroditsky (2003). Employing LLMs as a novel method for examining psycholinguistic biases related to grammatical gender, we prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. In particular, we look at adjective co-occurrences across gender and languages, and train a binary classifier to predict grammatical gender given adjectives an LLM uses to describe a noun. Surprisingly, we find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability. We show that while LLMs may describe words differently in different languages, they are biased similarly.
Paper Structure (20 sections, 3 equations, 4 figures, 6 tables)

This paper contains 20 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Describing nouns in different languages
  4. Predicting gender from descriptions
  5. Experiments
  6. Implementation details
  7. Bias in generated adjectives
  8. Do languages show similar biases?
  9. Predicting the gendered nouns
  10. Transfer between languages
  11. Discussion
  12. Reproducibility
  13. Importance of our results
  14. Conclusion
  15. Limitations
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Probing the bias of multilingual LLMs. We prompt a LLM to describe gendered nouns using adjectives. This allows us to study psycholinguistic biases of LLMs. For example, if the generated adjectives are predictive of the nouns's gender, we can, by training a binary classifier, predict grammatical gender by only looking at the adjectives a LLM uses to describe a word.
  • Figure 2: Bias when describing gendered nouns. Here we prompt an LLM in Spanish and for a random sample of adjectives, show the percentage of masculine nouns they were used for.
  • Figure 3: Gendered adjective similarity sccores.
  • Figure 4: Bias when describing gendered nouns. Here we prompt an LLM in Bulgarian, French, and German and for a random sample of adjectives, show the percentage of masculine nouns they were used for.