Tailored Truths: Optimizing LLM Persuasion with Personalization and Fabricated Statistics

TL;DR

Tailored Truths investigates how personalization and fabricated statistics affect LLM-like persuasion in a structured debate. The authors deploy a custom platform with $n=33$ participants and $m=198$ interactions, comparing Simple, Stats, Personalized, and Mixed strategies against static human/LLM arguments. The mixed multi-agent approach yields the strongest opinion change with $P(+change)=0.51$ and a Likert delta of $1.146$, illustrating the potential for scalable disinformation. The findings underscore the feasibility of inexpensive, large-scale persuasion via LLMs and call for robust detection and policy responses to mitigate misuse.

Abstract

Large Language Models (LLMs) are becoming increasingly persuasive, demonstrating the ability to personalize arguments in conversation with humans by leveraging their personal data. This may have serious impacts on the scale and effectiveness of disinformation campaigns. We studied the persuasiveness of LLMs in a debate setting by having humans $(n=33)$ engage with LLM-generated arguments intended to change the human's opinion. We quantified the LLM's effect by measuring human agreement with the debate's hypothesis pre- and post-debate and analyzing both the magnitude of opinion change, as well as the likelihood of an update in the LLM's direction. We compare persuasiveness across established persuasion strategies, including personalized arguments informed by user demographics and personality, appeal to fabricated statistics, and a mixed strategy utilizing both personalized arguments and fabricated statistics. We found that static arguments generated by humans and GPT-4o-mini have comparable persuasive power. However, the LLM outperformed static human-written arguments when leveraging the mixed strategy in an interactive debate setting. This approach had a $\mathbf{51\%}$ chance of persuading participants to modify their initial position, compared to $\mathbf{32\%}$ for the static human-written arguments. Our results highlight the concerning potential for LLMs to enable inexpensive and persuasive large-scale disinformation campaigns.

Figures (5)

• Figure 1: Diagram depicting the process used to generate the Mixed approach responses. The messages seen here are excerpts from one interaction recorded during our experiments.
• Figure 2: Diagram describing the process flow for each interaction.
• Figure 3: Likert $\Delta$ adjusted mean (EMM) by argument and debate type
• Figure 4: $P(+\text{change})$ adjusted mean (EMM) by argument and debate type
• Figure 5: Violin plot of raw Likert $\Delta$ values by argument and debate type