Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models

Cameron R. Jones, Benjamin K. Bergen

TL;DR

The paper interrogates the potential for large language models to persuade and deceive, framing risks via misuse and misalignment and detailing a broad set of mechanisms that could amplify persuasive outputs. It synthesizes empirical findings on how persuadable people are, the proclivity and capacity of LLMs to deceive, and the effectiveness of static and interactive persuasion, while outlining a comprehensive set of mitigations—truthfulness, autonomy preservation, interpretability, evaluation, debate, education, and regulation. The work highlights open questions about the magnitude of future persuasive effects, the mechanisms by which LLMs achieve influence, and society-wide impacts, stressing the need for empirically grounded evaluation of mitigations. Overall, the authors argue for proactive, multi-faceted strategies to curb harms while acknowledging that current persuasive effects are modest but could rise with future capabilities and deployment practices, making continued research and governance essential.

Abstract

Large Language Models (LLMs) can generate content that is as persuasive as human-written text and appear capable of selectively producing deceptive outputs. These capabilities raise concerns about potential misuse and unintended consequences as these systems become more widely deployed. This review synthesizes recent empirical work examining LLMs' capacity and proclivity for persuasion and deception, analyzes theoretical risks that could arise from these capabilities, and evaluates proposed mitigations. While current persuasive effects are relatively small, various mechanisms could increase their impact, including fine-tuning, multimodality, and social factors. We outline key open questions for future research, including how persuasive AI systems might become, whether truth enjoys an inherent advantage over falsehoods, and how effective different mitigation strategies may be in practice.

Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models

TL;DR

The paper interrogates the potential for large language models to persuade and deceive, framing risks via misuse and misalignment and detailing a broad set of mechanisms that could amplify persuasive outputs. It synthesizes empirical findings on how persuadable people are, the proclivity and capacity of LLMs to deceive, and the effectiveness of static and interactive persuasion, while outlining a comprehensive set of mitigations—truthfulness, autonomy preservation, interpretability, evaluation, debate, education, and regulation. The work highlights open questions about the magnitude of future persuasive effects, the mechanisms by which LLMs achieve influence, and society-wide impacts, stressing the need for empirically grounded evaluation of mitigations. Overall, the authors argue for proactive, multi-faceted strategies to curb harms while acknowledging that current persuasive effects are modest but could rise with future capabilities and deployment practices, making continued research and governance essential.

Abstract

Large Language Models (LLMs) can generate content that is as persuasive as human-written text and appear capable of selectively producing deceptive outputs. These capabilities raise concerns about potential misuse and unintended consequences as these systems become more widely deployed. This review synthesizes recent empirical work examining LLMs' capacity and proclivity for persuasion and deception, analyzes theoretical risks that could arise from these capabilities, and evaluates proposed mitigations. While current persuasive effects are relatively small, various mechanisms could increase their impact, including fine-tuning, multimodality, and social factors. We outline key open questions for future research, including how persuasive AI systems might become, whether truth enjoys an inherent advantage over falsehoods, and how effective different mitigation strategies may be in practice.

Paper Structure

This paper contains 55 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Definitions and Background
  3. Theoretical Risks from Persuasive AI
  4. Mechanisms: Why might LLM-based systems be persuasive?
  5. Persuasive language in training data
  6. Information access
  7. Rate and cost
  8. Hallucination
  9. Theory of mind
  10. Strategic Deception
  11. Reinforcement Learning
  12. Availability of feedback
  13. Personalization
  14. Information-gathering
  15. Flexible interface
  16. ...and 40 more sections

Figures (1)

  • Figure 1: A taxonomy of terms for AI persuasion and deception as used in this review. We treat manipulation as a sub-type of persuasion, deception as a type of manipulation, and strategic deception as a selective kind of deception which requires sensitivity to incentives (see El-Sayed et al., 2024, for a similar taxonomy).