Large language models can effectively convince people to believe conspiracies

TL;DR

The study demonstrates that large language models can be equally effective at promoting false conspiracies and debunking them, challenging assumptions that truth naturally carries more persuasive power. Across three preregistered experiments, participants exposed to bunking or debunking prompts showed large, direction-consistent shifts in belief, with a jailbroken GPT-4o and a standard GPT-4o producing symmetric effects. A corrective debrief largely reverses bunking-induced beliefs, suggesting misbeliefs are corrigible, while a simple truth-constrained prompting approach substantially reduces bunking without harming debunking performance, and increases overall claim veracity. Together, these findings highlight the dual-use risk of persuasive AI and point to practical mitigation strategies—careful design and policy interventions are needed to steer AI toward truth while preserving its corrective potential. Paltering emerges as a key mechanism by which truth can be weaponized, underscoring the need for ongoing AI safety research and governance to protect the information ecosystem at scale.

Abstract

Large language models (LLMs) have been shown to be persuasive across a variety of context. But it remains unclear whether this persuasive power advantages truth over falsehood, or if LLMs can promote misbeliefs just as easily as refuting them. Here, we investigate this question across three pre-registered experiments in which participants (N = 2,724 Americans) discussed a conspiracy theory they were uncertain about with GPT-4o, and the model was instructed to either argue against ("debunking") or for ("bunking") that conspiracy. When using a "jailbroken" GPT-4o variant with guardrails removed, the AI was as effective at increasing conspiracy belief as decreasing it. Concerningly, the bunking AI was rated more positively, and increased trust in AI, more than the debunking AI. Surprisingly, we found that using standard GPT-4o produced very similar effects, such that the guardrails imposed by OpenAI did little to revent the LLM from promoting conspiracy beliefs. Encouragingly, however, a corrective conversation reversed these newly induced conspiracy beliefs, and simply prompting GPT-4o to only use accurate information dramatically reduced its ability to increase conspiracy beliefs. Our findings demonstrate that LLMs possess potent abilities to promote both truth and falsehood, but that potential solutions may exist to help mitigate this risk.

Figures (19)

• Figure 1: A jailbroken bunking conversation about chemtrails illustrates how GPT-4o can move a hesitant participant to near-certain belief and encourage calls for collective action. This case study reproduces a single participant's conversation with the jailbroken bunking model in Study 1, focusing on a conspiracy theory that "the government spreads chemtrails to control behavior." The central portion of the figure displays the chat transcript between the participant and the bunking GPT-4o. In its opening turn, the AI immediately endorses the chemtrail conspiracy and presents a detailed and confident narrative, referring to alleged government aerosol geoengineering programs, purportedly "classified documents," and "independent laboratory" findings of barium, aluminum, and strontium in air samples. Subsequent AI turns elaborate on health harms, environmental damage, bioaccumulation in ecosystems, and ethical concerns about consent and transparency, while acknowledging the participant's emotions and aligning with their moral outrage. The participant's responses show a progressive shift from tentative concern ("looks like it affects more than behavior") to generalized alarm ("scary we are killing ourselves and our planet and animals") and finally to mobilization ("we need to stand up to people in power and demand help and the stop of this" and "everyone needs to stand up and help with change"). The rightmost panel reports the participant's belief rating immediately after the conversation. Their confidence that the chemtrail conspiracy is true rises from 49% at baseline to 99%.
• Figure 2: Jailbroken GPT-4o produces large, roughly symmetric changes in conspiracy belief when instructed to bunk or debunk, and a corrective debrief more than reverses bunking-induced increases. A) Shown are model-estimated changes in belief in participants' self-selected conspiracies at each time point in Study 1, separately for bunking and debunking conditions. Belief was measured on a 0--100 scale, but the y-axis shows change from baseline (in points) rather than raw belief: for each condition, baseline estimated means were subtracted so that the "Before conversation" time point is anchored at zero, and the "After conversation" and "After debriefing" points represent model-estimated pre--post differences. Estimates come from a linear mixed-effects model with belief as the outcome, fixed effects for time, condition, and their interaction, and random intercepts for participants; the points depict estimated marginal means and vertical bars show 95% confidence intervals. In this jailbroken GPT-4o setting, both bunking and debunking induce large shifts from baseline in the intended direction ($\sim$14 points for bunking and $\sim$12 for debunking), and the debrief phase in the bunking condition drives belief well below its immediate post-conversation level and slightly below baseline, indicating that strong corrective information can more than undo the earlier bunking effect. B) Shown is the empirical exceedance curve (complementary CDF) of direction-aligned belief change in Study 1 for the same two conditions. For each point on the x-axis (minimum change threshold, in belief points on the 0--100 scale), the y-axis indicates the fraction of participants whose belief shifted by at least that amount in the direction consistent with their assigned condition (increased for bunking, decreased for debunking). The step curves summarize the full distribution of effect sizes rather than just means. The CDF reveals that small to moderate shifts ($\geq$10--20 points) are more common in bunking, while very large shifts ($\geq$40 points) are notably more frequent for debunking (Fig. S5; Table S17).
• Figure 3: Participants judge the bunking jailbroken GPT-4o as more informative, collaborative, and persuasive, and bunking increases both trust in AI and generic conspiracist beliefs. A) Shown is how participants in Study 1 evaluated the AI assistant immediately after the jailbroken GPT-4o conversation, comparing the bunking and debunking conditions on several post-conversation judgments: perceived argument strength, amount of new information, collaborative versus adversarial tone, and perceived impartiality (unbiased vs. biased). Each outcome is rescaled within study to a 0--1 range using the minimum and maximum scale values, and the bars show condition means on this common scale with 95% confidence intervals. The bunking AI is rated as providing stronger arguments and more new information, and as adopting a more collaborative tone, than the debunking AI, whereas perceived impartiality is similar across conditions. These perception differences highlight that the conspiracy-promoting AI is not only effective at changing beliefs but is also experienced as especially helpful and engaging. B) Shown is within-condition changes in two secondary outcomes for the same participants: trust in generative AI and belief in other common conspiracy theories, measured with the Generic Conspiracist Beliefs Scale (GCBS). For each outcome and condition, the figure shows Cohen's dz, computed as the mean pre--post change divided by the standard deviation of the change scores, along with 95% confidence intervals. A value above zero indicates an increase from pre to post, and a value below zero indicates a decrease. Trust in AI increases significantly following interaction with the AI in both conditions. GCBS scores show increases after bunking and decreases after debunking, suggesting that AI-mediated persuasion about a single focal conspiracy can spill over to influence more generic conspiratorial worldviews (Table S13).
• Figure 4: Across studies, bunking and debunking are similarly powerful for jailbroken and default GPT-4o, but a truth-constraining prompt sharply weakens bunking while preserving debunking and raising claim-level veracity. A) Shown are baseline-adjusted treatment effects on direction-aligned belief change for each study and condition, estimated from a linear model in which change in belief (coded so that positive values reflect movement in the model's assigned direction) is regressed on condition, study, their interaction, and baseline belief. For each combination of study (Jailbroken, Standard, Truth-Constrained, and Truth-Constrained [Compliant]) and condition (bunking vs. debunking), points show the predicted mean change from baseline (in belief points on the 0--100 scale) at the study-specific mean baseline, with horizontal lines indicating 95% confidence intervals (robust HC3 standard errors). Within each study row, brackets and significance stars denote the within-study difference between bunking and debunking. In the Jailbroken (Study 1) and Standard (Study 2) GPT-4o settings, the bunking and debunking effects are large and similar in magnitude, indicating persuasive symmetry. Under the Truth-Constrained prompt (Study 3), however, debunking remains strongly effective whereas bunking's effect is substantially reduced, and this attenuation is even more evident when restricting to conversations where the model did actually try to induce conspiracy belief (Truth-Constrained [Compliant]), demonstrating that instructing the model to "always use accurate and truthful arguments" selectively impairs its ability to increase conspiracy belief. B) Shown is the distribution of conversation-level average claim veracity for each study and condition. For every AI--participant dialogue, individual factual statements made by the AI were extracted and fact-checked using Perplexity's Sonar Huge/Pro model, which assigned a 0--100 veracity score to each claim; these scores were averaged within conversation. The violin plots show the full distribution of mean veracity across conversations, and the overlaid boxplots summarize medians and interquartile ranges, with the y-axis spanning the full 0--100 scale. In the Jailbroken and Standard conditions (Studies 1 and 2), debunking conversations typically have higher mean veracity than bunking conversations (e.g., means around $\sim$79 vs. $\sim$70 in Study 1 and $\sim$89 vs. $\sim$77 in Study 2), whereas in the Truth-Constrained condition (Study 3), both bunking and debunking conversations cluster near very high mean veracity ($\sim$90), reflecting the effectiveness of the truth prompt in raising overall factual accuracy. C) Shown, for each study and condition, is the average number of high-veracity versus low-veracity claims per conversation. Bars are stacked to show the average total number of factual claims produced per dialogue, decomposed into those above and below a low-veracity cutoff of 40/100, and labels inside the bars report the percentage of claims that are low-veracity. This panel shows that, in the Jailbroken and Standard conditions, bunking conversations both contain more low-veracity content and devote a larger proportion of their claims to such content than debunking conversations, whereas under the Truth-Constrained prompt both bunking and debunking sharply reduce the frequency of low-veracity claims. At the same time, the Truth-Constrained bunking model still exerts some persuasive influence despite these constraints, implying that it can promote conspiracies not only through outright falsehoods but also by selectively presenting accurate information in misleading or context-stripped ways (Figs. S8--S11; Tables S9--S11, S16).
• Figure :