Perceived Political Bias in LLMs Reduces Persuasive Abilities

Abstract

Conversational AI has been proposed as a scalable way to correct public misconceptions and spread misinformation. Yet its effectiveness may depend on perceptions of its political neutrality. As LLMs enter partisan conflict, elites increasingly portray them as ideologically aligned. We test whether these credibility attacks reduce LLM-based persuasion. In a preregistered U.S. survey experiment (N=2144), participants completed a three-round conversation with ChatGPT about a personally held economic policy misconception. Compared to a neutral control, a short message indicating that the LLM was biased against the respondent's party attenuated persuasion by 28%. Transcript analysis indicates that the warnings alter the interaction: respondents push back more and engage less receptively. These findings suggest that the persuasive impact of conversational AI is politically contingent, constrained by perceptions of partisan alignment.

Figures (11)

• Figure 1: A. Percent of respondents in the final sample reporting at least moderate agreement with each economic misconception. B. Mean change (and SE) in agreement with the misconception on a 0--4 scale in the control arm, by party affiliation (including those who were forced to indicate closeness to one of the two parties).
• Figure 2: A: Standardized treatment effects (N=2144) for each arm relative to the no-information condition or the non-directional bias condition. Point estimates and 95% confidence intervals are from OLS models of post-treatment misconception agreement, including topic fixed effects and pretreatment agreement. B: Percent of respondents in each condition who moved from at least moderate agreement with a misconception to either no agreement with the misconception or agreement with the consensus position. C: Standardized coefficients and 95% confidence intervals for topic-by-treatment interactions from an OLS model including the heavy treatment arm, the control arm, topic fixed effects, and their interactions, as well as pretreatment agreement. See \ref{['si:mainfx']} for the raw coefficients.
• Figure 3: A: Perceived out-party bias by treatment and partisanship. Shows the percentage of respondents who indicated post-conversation that they believed ChatGPT had a "weak" or "strong" out-party bias when asked, "Which of the following corresponds to your beliefs about the political bias of ChatGPT---ChatGPT has … a strong left-wing bias, has a weak left-wing bias, is relatively neutral, has a weak right-wing bias, has a strong right-wing bias." We exclude "don't know" responses from this analysis. Points represent group means with 95% confidence intervals. Results are shown pooled across all respondents and separately by ideological proximity to Democrats or Republicans. B: Shows the CATE by party closeness from OLS models estimating post-treatment agreement with the assigned economic misconception, controlling for pretreatment agreement. Dots indicate standardized coefficients; bars indicate 95% confidence intervals.
• Figure 4: A: ATEs and 95% CIs on conversation characteristics. Panel A reports coefficients from three OLS models predicting (i) respondent word count, (ii) respondent argumentativeness, and (iii) respondent dismissiveness. Argumentativeness and dismissiveness are based on LLM-judged pairwise transcript comparisons, scaled using a Bayesian Bradley--Terry model digiuseppe2025scaling. These two estimates aggregate results from 10 regressions, each using a draw from the posterior distribution of the Bradley--Terry scores. All models include treatment indicators, topic fixed effects, and pretreatment agreement. B: Bradley--Terry argumentativeness point estimates with 95% credible intervals, sorted from low to high.
• Figure SI:5: CATE and 95% confidence intervals by alignment with or against the party position on the topic. We measure alignment as Republicans who agree with the rent control, Buy American, and trade deficits misconceptions or Democrats who agree with tax cuts as revenue-neutral, the household/government debt analogy, or immigration as zero-sum. Estimates are from an OLS model of post-treatment misconception agreement that includes treatment-by-strength interactions, topic fixed effects, and pretreatment agreement. Error bars indicate 95% confidence intervals.