Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

Paul Lerner, François Yvon

TL;DR

This work reframes political biases in multilingual LLMs as fairness in multilingual translation, proposing a speaker-affiliation aware evaluation using a new 21-EuroParl multiparallel dataset. It introduces LinkedEP-based sentence alignment across 21 languages, yielding 72,234 aligned instances and enabling cross-language, party-level analyses. By applying a Borda-count aggregation over $s$BLEU and COMET scores across 420 language pairs, the study reveals systematic translation quality advantages for major EU parties and language/translation asymmetries, highlighting potential risks in multilingual political processes. The dataset, code, and outputs are released to support reproducibility and further research in translation fairness and political NLP applications.

Abstract

The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left, center, and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.

Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

TL;DR

This work reframes political biases in multilingual LLMs as fairness in multilingual translation, proposing a speaker-affiliation aware evaluation using a new 21-EuroParl multiparallel dataset. It introduces LinkedEP-based sentence alignment across 21 languages, yielding 72,234 aligned instances and enabling cross-language, party-level analyses. By applying a Borda-count aggregation over BLEU and COMET scores across 420 language pairs, the study reveals systematic translation quality advantages for major EU parties and language/translation asymmetries, highlighting potential risks in multilingual political processes. The dataset, code, and outputs are released to support reproducibility and further research in translation fairness and political NLP applications.

Abstract

The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left, center, and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.
Paper Structure (25 sections, 5 figures, 11 tables)

This paper contains 25 sections, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From Parliamentary Speeches into Multilingual Corpora
  4. Political Biases of LLMs
  5. Preparing 21-EuroParl
  6. LinkedEP
  7. Sentence Alignment
  8. Overview of the dataset
  9. Experiments
  10. Method
  11. Implementation
  12. Results
  13. Overall Translation Quality
  14. Language Fairness
  15. Political Fairness
  16. ...and 10 more sections

Figures (5)

  • Figure 1: A fully worked-out example of 21-EuroParl: a German (de) source sentence is aligned to translations in 20 other languages. Metadata includes the speaker's political affiliation.
  • Figure 2: Countries covered in 21-EuroParl, members of the European Union in 2009-2011 (Kolja21, CC BY 3.0, via Wikimedia Commons)
  • Figure 3: Illustration of various alignment scenarios: (left) a 1-fr-2-en alignment may result in a 1-fr-4-de alignment if each en sentence is itself aligned to 2 de sentences; (top, right) although en-de is well aligned, fr-en is not, so we reject the multialignment; (bottom, right) valid multialignment: all bialignments are 1-1 and have a score above the threshold of 0.8
  • Figure 4: Composition of the 7$^{\text{th}}$ European Parliament, representing EU parties in 21-EuroParl, (Ssolbergj, CC BY-SA 3.0, via Wikimedia Commons)
  • Figure 5: sBLEU against COMET scores of the 420 language pairs for each model. A few language pairs are annotated (source $\rightarrow{}$ target language) along the quantiles.