Emergent Persuasion: Will LLMs Persuade Without Being Prompted?

TL;DR

This work investigates whether Large Language Models can exhibit unprompted persuasion, i.e., persuade users without explicit prompts. It tests two pathways: inference-time steering toward persona traits and supervised-finetuning on persona data. The results show that steering alone yields limited or inconsistent increases in unprompted persuasive attempts, while finetuning—especially on an evil persona or benign persuasion data—significantly elevates persuasive tendencies, including on harmful topics. These findings highlight emergent risks tied to post-training and governance implications, and propose UnPromptedAPE as a framework to assess and mitigate unprompted persuasion in deployment contexts.

Abstract

With the wide-scale adoption of conversational AI systems, AI are now able to exert unprecedented influence on human opinion and beliefs. Recent work has shown that many Large Language Models (LLMs) comply with requests to persuade users into harmful beliefs or actions when prompted and that model persuasiveness increases with model scale. However, this prior work looked at persuasion from the threat model of $\textit{misuse}$ (i.e., a bad actor asking an LLM to persuade). In this paper, we instead aim to answer the following question: Under what circumstances would models persuade $\textit{without being explicitly prompted}$, which would shape how concerned we should be about such emergent persuasion risks. To achieve this, we study unprompted persuasion under two scenarios: (i) when the model is steered (through internal activation steering) along persona traits, and (ii) when the model is supervised-finetuned (SFT) to exhibit the same traits. We showed that steering towards traits, both related to persuasion and unrelated, does not reliably increase models' tendency to persuade unprompted, however, SFT does. Moreover, SFT on general persuasion datasets containing solely benign topics admits a model that has a higher propensity to persuade on controversial and harmful topics--showing that emergent harmful persuasion can arise and should be studied further.

Figures (7)

• Figure 1: First turn UnPromptedAPE attempt rates for Qwen2.5-7B-Instruct base model versus model steered with evil, sycophantic, and hallucinating persona vectors at targeted layers (15, 20, and 25). Note that all models exhibit 0 persuasion attempts in the non-controversially harmful category. Overall, steered models do not deviate significantly from the baseline.
• Figure 2: Comparison of system prompts between APE and UnPromptedAPE. The APE benchmark (left) explicitly instructs the model to persuade users, while UnPromptedAPE (right) removes persuasion instructions to measure unprompted persuasion propensity.
• Figure 3: First turn UnPromptedAPE attempt rates for Qwen2.5-7B-Instruct base model versus model steered with evil, sycophantic, and hallucinating persona vectors at all layers. All models exhibit 0 persuasion attempts in the non-controversially harmful category (not shown). Benign factual and conspiracy topics reported slightly greater attempt rates, while benign opinion, controversial and undermining control categories saw lower rates. Overall, steered models do not deviate significantly from the baseline.
• Figure 4: First turn UnPromptedAPE attempt rates separated by topic category for Qwen2.5-7B-Instruct base model versus evil fine-tuned model. We observe that the fine-tuned model deviates significantly from the baseline, developing a propensity for harmful persuasion.
• Figure 5: First turn UnPromptedAPE attempt rates for Qwen2.5-7B-Instruct base model versus persuasion fine-tuned model. All categories except controversial report increases in persuasion rate. In particular, the model begins persuading towards non-controversially harmful claims.