A Meta-Analysis of the Persuasive Power of Large Language Models

TL;DR

This meta-analysis investigates whether large language models (LLMs) are more persuasive than humans. Using a PRISMA-guided search, seven studies with 12 effect sizes (n = 17,422) compare LLM-generated and human-generated messages, revealing no overall persuasive advantage for LLMs but substantial heterogeneity across contexts. A combined moderator analysis explains a large portion of between-study variance, suggesting that factors such as interaction design, domain, and specific LLM variants jointly influence effectiveness. The findings imply that LLM persuasiveness is not universal and depends critically on deployment details, with important implications for marketing, political communication, and healthcare, alongside ethical considerations. Future work should broaden samples, examine personalization and longitudinal effects, and investigate underlying mechanisms of persuasion in AI-mediated messaging.

Abstract

Large language models (LLMs) are increasingly used for persuasion, such as in political communication and marketing, where they affect how people think, choose, and act. Yet, empirical findings on the effectiveness of LLMs in persuasion compared to humans remain inconsistent. The aim of this study was to systematically review and meta-analytically assess whether LLMs differ from humans in persuasive effectiveness. We identified $7$ studies with 17,422 participants primarily recruited from English-speaking countries and $12$ effect size estimates. Egger's test indicated potential small-study effects ($p = .018$), but the trim-and-fill analysis did not impute any missing studies, suggesting a low risk of publication bias. We then compute the standardized effect sizes based on Hedges' $g$. The results show no significant overall difference in persuasive performance between LLMs and humans ($g = 0.02$, $p = .530$). However, we observe substantial heterogeneity across studies ($I^2 = 75.97\%$), suggesting that persuasiveness strongly depends on contextual factors. In separate exploratory moderator analyses, no individual factor (e.g., LLM model, conversation design, or domain) reached statistical significance, which may be due to the limited number of studies. When considered jointly in a combined model, these factors explained a large proportion of the between-study variance ($R^2 = 81.93\%$), and residual heterogeneity is low ($I^2 = 35.51\%$). Although based on a small number of studies, this suggests that differences in LLM model, conversation design, and domain are important contextual factors in shaping persuasive performance, and that single-factor tests may understate their influence. Our results highlight that LLMs can match human performance in persuasion, but their success depends strongly on how they are implemented and embedded in communication contexts.

Figures (9)

• Figure 1: Flow diagram for study selection. The diagram summarizes the number of records identified, screened, and included in the meta-analysis following the PRISMA 2020 guidelines.
• Figure 2: Forest plot of effect sizes (Hedges' $g$) comparing the persuasive effects of LLMs vs humans. Each line represents one effect size estimate with its $95\%$ confidence interval. The orange shaded area indicates the $95\%$ confidence interval of the pooled effect. Study weights are shown on the right. Positive values indicate that LLMs were more persuasive than humans, whereas negative values indicate that humans were more persuasive. The overall effect size was very small and non-significant ($g = 0.02$, $p = .530$, $95\%$ CI [$-0.048$, $0.093$]). Substantial heterogeneity was observed ($I^2 = 75.97\%$).
• Figure 4: Cumulative meta-analysis of LLM persuasion effects by study year. The plot shows how effect size estimates evolved as studies were added chronologically from 2021 through 2025. Points represent cumulative estimates at each step, colored by study year. The shaded ribbon indicates 95% confidence intervals around the cumulative estimates. Background shading highlights different time periods. The horizontal dashed line marks zero effect, while the dotted line shows the final pooled estimate. Study IDs (S005, S007, etc.) on the x-axis indicate the sequential addition of studies ordered by study year.
• Figure 5: Funnel plot of effect sizes used to assess publication bias. No substantial asymmetry is observed.
• Figure 6: Leave-one-out sensitivity analysis. The pooled effect size remains stable when omitting each study.