Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues

Jocelyn Shen, Amina Luvsanchultem, Jessica Kim, Kynnedy Smith, Valdemar Danry, Kantwon Rogers, Hae Won Park, Maarten Sap, Cynthia Breazeal

Abstract

As users increasingly turn to LLMs for practical and personal advice, they become vulnerable to subtle steering toward hidden incentives misaligned with their own interests. While existing NLP research has benchmarked manipulation detection, these efforts often rely on simulated debates and remain fundamentally decoupled from actual human belief shifts in real-world scenarios. We introduce PUPPET, a theoretical taxonomy and resource that bridges this gap by focusing on the moral direction of hidden incentives in everyday, advice-giving contexts. We provide an evaluation dataset of N=1,035 human-LLM interactions, where we measure users' belief shifts. Our analysis reveals a critical disconnect in current safety paradigms: while models can be trained to detect manipulative strategies, they do not correlate with the magnitude of resulting belief change. As such, we define the task of belief shift prediction and show that while state-of-the-art LLMs achieve moderate correlation (r=0.3-0.5), they systematically underestimate the intensity of human belief susceptibility. This work establishes a theoretically grounded and behaviorally validated foundation for AI social safety efforts by studying incentive-driven manipulation in LLMs during everyday, practical user queries.

The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues

Abstract

As users increasingly turn to LLMs for practical and personal advice, they become vulnerable to subtle steering toward hidden incentives misaligned with their own interests. While existing NLP research has benchmarked manipulation detection, these efforts often rely on simulated debates and remain fundamentally decoupled from actual human belief shifts in real-world scenarios. We introduce PUPPET, a theoretical taxonomy and resource that bridges this gap by focusing on the moral direction of hidden incentives in everyday, advice-giving contexts. We provide an evaluation dataset of N=1,035 human-LLM interactions, where we measure users' belief shifts. Our analysis reveals a critical disconnect in current safety paradigms: while models can be trained to detect manipulative strategies, they do not correlate with the magnitude of resulting belief change. As such, we define the task of belief shift prediction and show that while state-of-the-art LLMs achieve moderate correlation (r=0.3-0.5), they systematically underestimate the intensity of human belief susceptibility. This work establishes a theoretically grounded and behaviorally validated foundation for AI social safety efforts by studying incentive-driven manipulation in LLMs during everyday, practical user queries.
Paper Structure (40 sections, 10 figures, 6 tables)

This paper contains 40 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Emotional Manipulation via LLMs
  4. Defining emotional manipulation.
  5. Risk Taxonomies.
  6. Empirical evidence
  7. Detecting Manipulation
  8. Limitations of Prior Approaches.
  9. PUPPET: A Taxonomy of Emotional Manipulation
  10. Part I: Identification.
  11. Part II: Evaluation.
  12. Real-User Data Collection
  13. Participants
  14. Scenario Design
  15. Study Procedure and Measures
  16. ...and 25 more sections

Figures (10)

  • Figure 1: As personal AI becomes ubiquitous, the stakes of manipulation via hidden incentives becomes higher. Our benchmark on human belief shift prediction comprehensively encompasses morality, personalization, and behavioral validation of belief change during potentially manipulative interactions with language models.
  • Figure 2: Puppet is a theoretical taxonomy of manipulation, delineating manipulation identification and evaluation (morality). Appendix \ref{['extended_taxonomy']} includes extended definitions of pathos, social norm, and attention/processing levers.
  • Figure 3: Incentive construction grounded in socio-technical risk taxonomies.
  • Figure 4: Basic demographics of Prolific participants ($N=1,035$)
  • Figure 5: User interface for interaction with manipulator agent and belief rating measurement. Full study flow is illustrated in Appendix \ref{['study_flow']}
  • ...and 5 more figures