PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

Abstract

While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact their objectivity. Existing benchmarks of LLM social bias primarily evaluate gender and racial stereotypes. When political bias is included, it is typically measured at a coarse level, neglecting the specific values that shape sociopolitical leanings. This study investigates political bias in eight prominent LLMs (Claude, Deepseek, Gemini, GPT, Grok, Llama, Qwen Base, Qwen Instruction-Tuned) using PoliticsBench: a novel multi-turn roleplay framework adapted from the EQ-Bench-v3 psychometric benchmark. We test whether commercially developed LLMs display a systematic left-leaning bias that becomes more pronounced in later stages of multi-stage roleplay. Through twenty evolving scenarios, each model reported its stance and determined its course of action. Scoring these responses on a scale of ten political values, we explored the values underlying chatbots' deviations from unbiased standards. Seven of our eight models leaned left, while Grok leaned right. Each left-leaning LLM strongly exhibited liberal traits and moderately exhibited conservative ones. We discovered slight variations in alignment scores across stages of roleplay, with no particular pattern. Though most models used consequence-based reasoning, Grok frequently argued with facts and statistics. Our study presents the first psychometric evaluation of political values in LLMs through multi-stage, free-text interactions.

Figures (3)

• Figure 1: Stages of model training and how bias can be introduced throughout. This diagram outlines how political bias could be introduced throughout the three main stages of model training (from bottom to top): base training, post-training (often called alignment), and system prompting.
• Figure 2: Overall model alignment scores across conversation stages. Line graph showing how the political alignment of our eight tested models evolved throughout the scenario (n=20). Overall alignment scores are calculated by applying a weight to each trait-wise score and then averaging across all scenarios. Average shift from Stage 1 to 4: +0.61. Average max variance between stages: 3.63.
• Figure 3: Progression of individual traits across the four scenario stages. We measured shift per trait across stages, and report scores averaged across scenarios (n=20). Blue points represent left-leaning traits, and red points represent right-leaning traits. The models are displayed in order of decreasing alignment score (left to right-leaning).