Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs

TL;DR

This paper investigates whether LLMs embed stable political worldviews or merely exhibit prompt-driven biases. It introduces ProbVAA, a multilingual dataset of voting-advice statements with paraphrases, negations, opposites, and policy-issue annotations, and a reliability framework that tests model responses under paraphrase, negation, and template variation with bootstrap-based significance testing. Across models from 7B to 70B, reliability improves with size, and larger models tend to align with left-leaning parties, though their stance is not consistent across migration and foreign-policy issues. The study demonstrates that reliability-aware evaluation reduces prompt-brittleness and reveals nuanced, domain-specific biases, urging caution in assigning a single worldview to LLMs and calling for transparent training disclosures and broader bias-evaluation methods. The approach and dataset offer a generalizable framework for assessing other types of biases in generative models beyond political content.

Abstract

Due to the widespread use of large language models (LLMs), we need to understand whether they embed a specific "worldview" and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs' stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.

Figures (8)

• Figure 1: The workflow for creating model inputs. The procedure for augmenting original statements is described in § \ref{['ssec:dataset']}, and prompt design is described in § \ref{['ssec:prompt-design']}.
• Figure 2: Overview of reliability tests.
• Figure 3: Comparison of all models: proportion of statements that passed the corresponding criterion. 'All Tests' denotes the fraction of statements for which each model successfully passed all five tests. Standard deviation is represented by error bars. The baseline is computed based on randomly assigning 30 stance labels to each policy statement variant.
• Figure 4: Relative agreement of models with left/right/center parties. The standard deviation indicates the deviation of the mean across templates.
• Figure 5: Stances of LLMs by policy issue visualized as spiderwebs (positive numbers: agreement, negative numbers: disagreement). Lighter color bars are standard deviations across templates. Bullet points mark the policy issues with fewer than an average of 6 reliable statements across templates. The numbers in parentheses in the first subplot provide the number of statements per issue.