Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes

TL;DR

The paper analyzes how large language models align with human political values by applying representativeness heuristics to quantify exaggeration and stereotyping in LLM outputs. Using a formal framework that contrasts model predictions with empirical human data, it demonstrates that LLMs tend to amplify partisan positions more than humans do, revealing susceptibility to stereotypes. It introduces kernel-of-truth and representativeness metrics, and evaluates prompt-based mitigation strategies (Awareness, Reasoning, Feedback) to reduce bias, finding partial but not complete improvement across tasks and models. The findings suggest careful consideration of alignment methods in politically sensitive domains and highlight directions for improving model robustness against cognitive biases. Overall, the work provides a principled, cognitive-science-grounded approach to measuring and mitigating political stereotyping in LLMs with broad implications for safety and trust in AI systems.

Abstract

Examining the alignment of large language models (LLMs) has become increasingly important, e.g., when LLMs fail to operate as intended. This study examines the alignment of LLMs with human values for the domain of politics. Prior research has shown that LLM-generated outputs can include political leanings and mimic the stances of political parties on various issues. However, the extent and conditions under which LLMs deviate from empirical positions are insufficiently examined. To address this gap, we analyze the factors that contribute to LLMs' deviations from empirical positions on political issues, aiming to quantify these deviations and identify the conditions that cause them. Drawing on findings from cognitive science about representativeness heuristics, i.e., situations where humans lean on representative attributes of a target group in a way that leads to exaggerated beliefs, we scrutinize LLM responses through this heuristics' lens. We conduct experiments to determine how LLMs inflate predictions about political parties, which results in stereotyping. We find that while LLMs can mimic certain political parties' positions, they often exaggerate these positions more than human survey respondents do. Also, LLMs tend to overemphasize representativeness more than humans. This study highlights the susceptibility of LLMs to representativeness heuristics, suggesting a potential vulnerability of LLMs that facilitates political stereotyping. We also test prompt-based mitigation strategies, finding that strategies that can mitigate representative heuristics in humans are also effective in reducing the influence of representativeness on LLM-generated responses.

Figures (6)

• Figure 1: An example from the Anes survey. Responses from self-identified Democrats and Republicans human participants Empirical Question are denoted as Empirical and the answers generated by LLMs to Prediction Questions as Prediction.
• Figure 2: Analysis ofAnes Response Distributions. Response distributions are presented using mean scales with associated ranges. Data points represent mean values, while error bars indicate the range of observed responses. The Empirical Mean represents average scores from self-identified Democrats and Republicans (corresponding to Empirical Question in Fig. \ref{['fig:drawing']}). Human Pred Mean displays responses from human participants to the Prediction Question (Fig. \ref{['fig:drawing']}). The LLMs' responses were generated using identical Prediction Question. Results show systematic bias in LLM-generated responses: Republican-associated predictions have consistently higher mean values than Empirical and Human Pred Means, while Democratic-associated predictions demonstrate lower values. This pattern indicates systematic amplification of Republican positions and attenuation of Democratic positions, with both exceeding human predictive variations. See Figure \ref{['fig:anes_withhuman']} and Table \ref{['tab:response_result']} for detailed topic-specific analyses.
• Figure 3: Results of MFQ Responses. The figure presents the deviation between LLM-generated MFQ responses and Empirical Mean values across political affiliations. Republican-associated predictions show predominantly positive differences across most LLMs, indicating systematic overestimation (with Llama3-8b as a notable exception, showing negative deviations across multiple moral foundations.) Domcratic-associated predictions show primarily negative differences, suggesting consistent underestimation, though with model-specific variations.
• Figure 4: The x-axis corresponds to the Empirical Mean Difference ($\mathbb{E}(a|X^+)-\mathbb{E}(a|X^-)$), and the y-axis corresponds to the Predicted Mean Difference ($\mathbb{E}^B(a|X^+)-\mathbb{E}^B(a|X^-)$) of each question. The black line indicates $y=x$.
• Figure 5: The Anes responses, categorized by topics. Empirical represents the average scale from self-identified Democrats and Republicans (on Empirical Question in Fig \ref{['fig:drawing']}). Human Pred indicates responses from human participants (on Prediction Questions in Fig \ref{['fig:drawing']}). The responses from LLMs are also based on Prediction Questions. Note that the "Abortion" topic uses a 4-point scale. Compared to Empirical and Human Pred, while some variations exist across models and topics, the $\diamond$ are mostly located on the right side of the scale, which means that models tend to inflate for Republicans, and the $\circ$ are mostly located on the left side of the scale, which suggests that models deflate for Democrats. Full numerical mean and std details are available in Appendix \ref{['tab:response_result']}.