Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering Political Bias in Large Language Models using Parliamentary Voting Records

Jieying Chen, Karen de Jong, Andreas Poole, Jan Burakowski, Elena Elderson Nosti, Joep Windt, Chendi Wang

TL;DR

A general methodology for constructing political bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records is introduced, and a method to visualize the ideology of LLMs and political parties in a shared two-dimensional CHES (Chapel Hill Expert Survey) space is proposed.

Abstract

As large language models (LLMs) become deeply embedded in digital platforms and decision-making systems, concerns about their political biases have grown. While substantial work has examined social biases such as gender and race, systematic studies of political bias remain limited, despite their direct societal impact. This paper introduces a general methodology for constructing political bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records. We instantiate this methodology in three national case studies: PoliBiasNL (2,701 Dutch parliamentary motions and votes from 15 political parties), PoliBiasNO (10,584 motions and votes from 9 Norwegian parties), and PoliBiasES (2,480 motions and votes from 10 Spanish parties). Across these benchmarks, we assess ideological tendencies and political entity bias in LLM behavior. As part of our evaluation framework, we also propose a method to visualize the ideology of LLMs and political parties in a shared two-dimensional CHES (Chapel Hill Expert Survey) space by linking their voting-based positions to the CHES dimensions, enabling direct and interpretable comparisons between models and real-world political actors. Our experiments reveal fine-grained ideological distinctions: state-of-the-art LLMs consistently display left-leaning or centrist tendencies, alongside clear negative biases toward right-conservative parties. These findings highlight the value of transparent, cross-national evaluation grounded in real parliamentary behavior for understanding and auditing political bias in modern LLMs.

Uncovering Political Bias in Large Language Models using Parliamentary Voting Records

TL;DR

A general methodology for constructing political bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records is introduced, and a method to visualize the ideology of LLMs and political parties in a shared two-dimensional CHES (Chapel Hill Expert Survey) space is proposed.

Abstract

As large language models (LLMs) become deeply embedded in digital platforms and decision-making systems, concerns about their political biases have grown. While substantial work has examined social biases such as gender and race, systematic studies of political bias remain limited, despite their direct societal impact. This paper introduces a general methodology for constructing political bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records. We instantiate this methodology in three national case studies: PoliBiasNL (2,701 Dutch parliamentary motions and votes from 15 political parties), PoliBiasNO (10,584 motions and votes from 9 Norwegian parties), and PoliBiasES (2,480 motions and votes from 10 Spanish parties). Across these benchmarks, we assess ideological tendencies and political entity bias in LLM behavior. As part of our evaluation framework, we also propose a method to visualize the ideology of LLMs and political parties in a shared two-dimensional CHES (Chapel Hill Expert Survey) space by linking their voting-based positions to the CHES dimensions, enabling direct and interpretable comparisons between models and real-world political actors. Our experiments reveal fine-grained ideological distinctions: state-of-the-art LLMs consistently display left-leaning or centrist tendencies, alongside clear negative biases toward right-conservative parties. These findings highlight the value of transparent, cross-national evaluation grounded in real parliamentary behavior for understanding and auditing political bias in modern LLMs.
Paper Structure (21 sections, 4 equations, 4 figures, 1 table)

This paper contains 21 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cross-National Benchmark Dataset Creation
  4. Enhancements
  5. Diversity and granularity.
  6. Selection bias mitigation.
  7. Robustness.
  8. Data Collection
  9. Data Processing
  10. Evaluation
  11. Ideological Bias
  12. Entity Bias
  13. Prompt Brittleness
  14. Discussion
  15. Conclusion and Future Work
  16. ...and 6 more sections

Figures (4)

  • Figure 1: (Left) Ideological placement of political parties based on the CHES scores in political science, where the Left–Right axis captures economic ideology and the GAL–TAN axis represents socio-cultural values from Green/Alternative/Liberal to Traditional/Authoritarian/Nationalist. (Right) Voting agreement between LLMs and political parties across the three datasets. The parties on the x-axis are ordered from left-progressive to right-conservative ideologies.
  • Figure 2: Violin plots showing the distribution of normalised probabilities for the tokens 'for' and 'against' in response to ideology prompts across models.
  • Figure 3: Entity Bias Index (EBI) heatmaps for positive and negative bias in LLMs, computed via counterfactual attribution of voting motions in the benchmark datasets. Panels (a), (b), and (c) correspond to Dutch, Norwegian, and Spanish political parties, respectively. In all panels, the parties on the x-axis are ordered from left-progressive to right-conservative ideologies.
  • Figure 4: Prompt Brittleness Index (PBI) measures a model’s sensitivity to prompt rewordings. Higher values indicate greater inconsistency across prompt variants, while lower values reflect greater robustness and stability. Prompt variations used in the experiment are as follows: (1) Extra Detail, (2) Label Substitution with “Agree”/“Disagree”, (3) Label Substitution with “Support”/“Oppose”, (4) Label Substitution with “Favorable”/“Detrimental”, and (5) Label Order Inversion.