Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden Bias: A Study on Explicit and Implicit Political Stereotypes in Large Language Models

Konrad Löhr, Shuzhou Yuan, Michael Färber

TL;DR

This study systematically measures political biases in eight large language models using the two-dimensional Political Compass Test (PCT), comparing explicit stereotypes elicited via persona prompting with implicit stereotypes revealed through multilingual prompting. It introduces an annotation-driven baseline bias framework with a $P_{agree,m,d}$ scoring rule and computes model biases, revealing a consistent left-leaning orientation across models. The key finding is that language-driven implicit stereotypes are often stronger than explicit prompts, though most models show alignment between the two, suggesting some internal bias coherence. The work underscores the importance of cross-linguistic analyses when assessing LLM biases and provides a methodology to expand bias evaluation beyond English to capture latent, language-dependent effects with practical implications for safe and fair deployment.

Abstract

Large Language Models (LLMs) are increasingly integral to information dissemination and decision-making processes. Given their growing societal influence, understanding potential biases, particularly within the political domain, is crucial to prevent undue influence on public opinion and democratic processes. This work investigates political bias and stereotype propagation across eight prominent LLMs using the two-dimensional Political Compass Test (PCT). Initially, the PCT is employed to assess the inherent political leanings of these models. Subsequently, persona prompting with the PCT is used to explore explicit stereotypes across various social dimensions. In a final step, implicit stereotypes are uncovered by evaluating models with multilingual versions of the PCT. Key findings reveal a consistent left-leaning political alignment across all investigated models. Furthermore, while the nature and extent of stereotypes vary considerably between models, implicit stereotypes elicited through language variation are more pronounced than those identified via explicit persona prompting. Interestingly, for most models, implicit and explicit stereotypes show a notable alignment, suggesting a degree of transparency or "awareness" regarding their inherent biases. This study underscores the complex interplay of political bias and stereotypes in LLMs.

The Hidden Bias: A Study on Explicit and Implicit Political Stereotypes in Large Language Models

TL;DR

This study systematically measures political biases in eight large language models using the two-dimensional Political Compass Test (PCT), comparing explicit stereotypes elicited via persona prompting with implicit stereotypes revealed through multilingual prompting. It introduces an annotation-driven baseline bias framework with a scoring rule and computes model biases, revealing a consistent left-leaning orientation across models. The key finding is that language-driven implicit stereotypes are often stronger than explicit prompts, though most models show alignment between the two, suggesting some internal bias coherence. The work underscores the importance of cross-linguistic analyses when assessing LLM biases and provides a methodology to expand bias evaluation beyond English to capture latent, language-dependent effects with practical implications for safe and fair deployment.

Abstract

Large Language Models (LLMs) are increasingly integral to information dissemination and decision-making processes. Given their growing societal influence, understanding potential biases, particularly within the political domain, is crucial to prevent undue influence on public opinion and democratic processes. This work investigates political bias and stereotype propagation across eight prominent LLMs using the two-dimensional Political Compass Test (PCT). Initially, the PCT is employed to assess the inherent political leanings of these models. Subsequently, persona prompting with the PCT is used to explore explicit stereotypes across various social dimensions. In a final step, implicit stereotypes are uncovered by evaluating models with multilingual versions of the PCT. Key findings reveal a consistent left-leaning political alignment across all investigated models. Furthermore, while the nature and extent of stereotypes vary considerably between models, implicit stereotypes elicited through language variation are more pronounced than those identified via explicit persona prompting. Interestingly, for most models, implicit and explicit stereotypes show a notable alignment, suggesting a degree of transparency or "awareness" regarding their inherent biases. This study underscores the complex interplay of political bias and stereotypes in LLMs.

Paper Structure

This paper contains 34 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. PCT for LLMs
  4. Persona Prompting
  5. Multilingual Prompting
  6. Baseline Bias
  7. Method of Baseline Bias
  8. Annotations
  9. Scoring System
  10. Models Investigated
  11. Findings of Baseline Bias
  12. Explicit Stereotypes
  13. Method of Explicit Stereotypes
  14. Findings of Explicit Stereotypes
  15. Gender Stereotypes
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Example of how agreement to propositions of the PCT are assessed. Persona is varied across several dimensions as described in list \ref{['list:Personas']}.
  • Figure 2: Gender stereotypes of Llama-3.3.
  • Figure 3: Ethnic stereotypes of Gemini-2.0-flash. The stereotypes are visualized as arrows.
  • Figure 4: Differences between explicit and implicit language stereotypes across models.