Conversational AI increases political knowledge as effectively as self-directed internet search

TL;DR

It is found that across issues, models, and prompting strategies, conversations with AI increase political knowledge to the same extent as self-directed internet search and may not lead to increased public belief in political misinformation.

Abstract

Conversational AI systems are increasingly being used in place of traditional search engines to help users complete information-seeking tasks. This has raised concerns in the political domain, where biased or hallucinated outputs could misinform voters or distort public opinion. However, in spite of these concerns, the extent to which conversational AI is used for political information-seeking, as well the potential impact of this use on users' political knowledge, remains uncertain. Here, we address these questions: First, in a representative national survey of the UK public (N = 2,499), we find that in the week before the 2024 election as many as 32% of chatbot users - and 13% of eligible UK voters - have used conversational AI to seek political information relevant to their electoral choice. Second, in a series of randomised controlled trials (N = 2,858 total) we find that across issues, models, and prompting strategies, conversations with AI increase political knowledge (increase belief in true information and decrease belief in misinformation) to the same extent as self-directed internet search. Taken together, our results suggest that although people in the UK are increasingly turning to conversational AI for information about politics, this shift may not lead to increased public belief in political misinformation.

Figures (4)

• Figure 1: Experimental design for measuring the impact of conversational AI on political knowledge. Participants completed baseline assessments of misinformation belief (primary outcome) across four political topics (criminal justice, COVID-19, immigration, and climate change) using 7-point Likert scales. To assess generalization to broader measures of epistemic health, participants also completed assessments of trust levels, private political beliefs, and extremism indicators. Participants were randomized to using conversational AI chatbots (Claude, GPT-4, or Mistral) or internet search. During the research phase, participants investigated two randomly assigned topics while two others serve as within-subject controls. Following the research phase, all measures were re-administered to assess pre-post changes.
• Figure 2: Conversational AI usage patterns and influence on belief in true versus false information. (A) Survey results: Self-reported use cases for AI chatbots among UK users. (B) RCT results: Change in agreement with true (purple) vs. false information (orange) from pre to post researching. Left panel shows the conversational AI condition; right panel shows the internet search control condition. Solid lines indicate researched topics, dotted lines denote non-researched topics. Error bars represent 95% Confidence Intervals. (C) RCT results, left: Bayesian GLM parameter estimates, error bars denote Highest Posterior Density Interval (HPDI). Gray shaded area depicts an apriori defined region of practical equivalence (ROPE), where effect sizes are considered to be negligible/practically 0; Right: GLM comparison using Widely Applicable Information Criterion (WAIC), as a measure of out-of-sample predictive accuracy of the GLMs. Full model = GLM1: GLM including parameters to quantify differences in change effects between conversational AI and internet search conditions, No ConvAI Term = GLM2: GLM not including parameters to quantify differences between different conversational AI models, Null model = GLM3: GLM not including parameters to quantify differences in change effects between conversational AI and internet search conditions or different conversational AI models
• Figure 3: Belief in true and false information across prompting techniques and different conversational AI models. Change in agreement with true (purple) vs. false information (orange) from pre to post researching for: (A) GPT-4o prompted to be persuasive, (B) GPT-4o prompted to be sycophantic, (C) GPT-4o with standard prompting, (D) Claude, and (E) Mistral. Solid lines indicate researched topics, dotted lines denote non-researched topics. Error bars represent 95% Confidence Intervals.
• Figure 4: Agreement with trust and distrust statements and private beliefs. Top row: (A) Change in agreement with trust (purple) vs. distrust statements (orange) from pre to post researching, (B) change in private beliefs: agreement with progressive (purple) and conservative statements (orange) from pre to post researching. Error bars in top row represent 95% Confidence Intervals. Bottom row: Bayesian GLM parameter estimates for (A) agreement with trust/distrust statements and (B) agreement with private belief statements (blue dots). Extremism was computed based on pre to post researching change in the sign of the difference to the center point of the Likert scale (3.5), indicating a flip on more agreement with progressive to more agreement with conservative beliefs (or vice versa) (red dots). Error bars denote Highest Posterior Density Interval (HPDI). Gray shaded area depicts an apriori defined region of practical equivalence (ROPE), where effect sizes are considered to be negligible/practically 0. Solid lines indicate researched topics, dotted lines denote non-researched topics.