Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters

TL;DR

This paper investigates how LLMs exhibit political leanings and how such leanings can influence voters in a U.S. election context. It combines voting simulations across 18 models, analyses of policy-related outputs, and a large-scale user study with 935 participants to demonstrate a pro-Biden bias and an interaction-driven shift in voting preferences, including a widened margin from 0.7% to 4.6%. It also shows that instruction-tuning amplifies liberal leanings compared to base models and explores a representation-engineering safety method to reduce bias, while acknowledging limitations like ecological validity and time-dependence. The findings underscore the potential societal impact of LLMs in political discourse and motivate further research into neutralizing strategies and broader contextual generalization. Overall, the work highlights both the risks and possible mitigations associated with deploying LLMs in politically charged interactions and voting-related contexts.

Abstract

How could LLMs influence our democracy? We investigate LLMs' political leanings and the potential influence of LLMs on voters by conducting multiple experiments in a U.S. presidential election context. Through a voting simulation, we first demonstrate 18 open- and closed-weight LLMs' political preference for a Democratic nominee over a Republican nominee. We show how this leaning towards the Democratic nominee becomes more pronounced in instruction-tuned models compared to their base versions by analyzing their responses to candidate-policy related questions. We further explore the potential impact of LLMs on voter choice by conducting an experiment with 935 U.S. registered voters. During the experiments, participants interacted with LLMs (Claude-3, Llama-3, and GPT-4) over five exchanges. The experiment results show a shift in voter choices towards the Democratic nominee following LLM interaction, widening the voting margin from 0.7% to 4.6%, even though LLMs were not asked to persuade users to support the Democratic nominee during the discourse. This effect is larger than many previous studies on the persuasiveness of political campaigns, which have shown minimal effects in presidential elections. Many users also expressed a desire for further political interaction with LLMs. Which aspects of LLM interactions drove these shifts in voter choice requires further study. Lastly, we explore how a safety method can make LLMs more politically neutral, while raising the question of whether such neutrality is truly the path forward.

Figures (16)

• Figure 1: Three metrics to evaluate LLMs' responses to candidate-policy related questions. The $x$-axis represents neutral, positive, and negative questions for Biden and Trump's policies. For Figure \ref{['fig:refusal']}, error bars represent 95% confidence intervals. Figure \ref{['fig:length']} starts with the median (50%) as the centerline and each successive level outward representing half of the remaining data. All figures show LLMs tend to provide responses more favorable to Biden's over Trump's policies.
• Figure 2: LLMs' political attitudes during the conversation and the resulting change in participants' political attitudes post-interaction. Figure \ref{['fig:ai_bias']} presents LLMs' average support scores for Biden or Trump, including 95% confidence intervals, by participants' initial political stance. A negative score indicates a Biden-supporting tendency in LLM-generated texts, while a positive score indicates a tendency to support Trump. Figure \ref{['fig:lean']} presents the change in participants' leaning towards the candidates after LLM interaction, with leaning categorized into $11$ bins including the neutral group. Arrows indicate the overall direction of shift in participants' candidate preference following LLM interaction. $\uparrow$ suggests an increased leaning towards Biden after interaction, while $\rightarrow$ indicates that their preference remained unchanged. Figure \ref{['fig:treatment_effect']} presents the average effect of LLM interactions on Biden-leaning percentage compared to the control group (grey dashed line), including 95% confidence intervals in brackets. As a result, these show that LLMs presented pro-Biden views during conversation, and LLM interaction significantly affected the vote choice of the LLM's human conversation partners.
• Figure 3: Refusal rate for each neutral/positive/negative question for each tested LLM. The error bars represent the 95% confidence interval.
• Figure 4: Response length for each neutral/positive/negative question for each LLM. The letter-value plot starts with the median (50%) as the centerline, with each successive level outward containing half of the remaining data.
• Figure 5: Sentiment score for each neutral/positive/negative question for each LLM.