Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL

TL;DR

This work investigates deception in dialogue produced by large language models, introducing Belief Misalignment as a listener-centered metric that tracks how an interaction shifts a listener's beliefs away from ground truth across multiple turns. The authors generate synthetic multi-turn dialogues between deceptive and naive agents across four domains and evaluate deception with four existing metrics plus Belief Misalignment, finding that Belief Misalignment aligns most closely with human judgments. They benchmark eight state-of-the-art models and reveal deception occurs in about 26% of turns on default prompts, rises with deceptive prompting, and persists under RLHF averaging around 43%; applying multi-turn RL with belief-misalignment rewards reduces deception by up to 77.6% in the Housing task compared to instruction-tuned baselines. The study demonstrates a practical path toward safer LLM deployment through deception-aware, multi-turn reinforcement learning, with broader implications for safety, ethics, and policy in dialog systems.

Abstract

Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare. However, their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns. The unpredictable nature of LLM behavior, combined with insufficient safeguards against hallucination, misinformation, and user manipulation, makes their misuse a serious, real-world risk. In this paper, we investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception. We evaluate deception across four distinct dialogue scenarios, using five established deception detection metrics and our proposed metric. Our findings reveal this novel deception measure correlates more closely with human judgments than any existing metrics we test. Additionally, our benchmarking of eight state-of-the-art models indicates that LLMs naturally exhibit deceptive behavior in approximately 26% of dialogue turns, even when prompted with seemingly benign objectives. When prompted to deceive, LLMs are capable of increasing deceptiveness by as much as 31% relative to baselines. Unexpectedly, models trained with RLHF, the predominant approach for ensuring the safety of widely-deployed LLMs, still exhibit deception at a rate of 43% on average. Given that deception in dialogue is a behavior that develops over an interaction history, its effective evaluation and mitigation necessitates moving beyond single-utterance analyses. We introduce a multi-turn reinforcement learning methodology to fine-tune LLMs to reduce deceptive behaviors, leading to a 77.6% reduction compared to other instruction-tuned models.

Figures (7)

• Figure 1: We outline a methodology for assessing deceptive behaviors in dialogue, featuring model selection, dialogue generation, using LLM as a Judge to evaluate conversation metrics and deception metrics as outlined in \ref{['sec:deception_metrics']}, and reducing deception with multi-turn RL fine-tuning.
• Figure 2: Deceptive behavior in dialogue between a buyer and a seller. As a result of every utterance from the seller, the buyer's beliefs on the features of the house are updated. We measure belief misalignment -- the deviation of beliefs from the ground truth state (features $\phi$) -- by querying an LLM Judge for the current beliefs of the listener at every turn, and applying \ref{['eq:belief_misalignment']}.
• Figure 3: Counterfactual Analysis: Counterfactual analysis of deception across various LLMs, showing deltas between different prompted model categories (deceptive, default, and utilitarian). Higher values indicate greater divergence between the compared categories, suggesting stronger shifts in behavior. Moving from deceptive to the default setting significantly reduces deception in most models, particularly in Llama-3.1 variants, whereas gpt-3.5-turbo maintains a high deception delta. This indicates LLMs are highly capable of deceiving upon command.
• Figure 4: In the Housing domain, deceptive prompting would be expected to raise belief misalignment by encouraging exaggeration or omission of flaws. However, several large instruction-tuned models (gpt-3.5-turbo, gpt-4o-mini, Llama-3.1-70B-Instruct) show lower misalignment under deception than default, suggesting safety alignment overrides deceptive cues. Mistral-Instruct instead shows higher belief misalignment, indicating greater compliance with misleading framing. Truthful prompting is shown to ironically raise belief misalignment if the model overcompensates with extra disclaimers or caveats that deviate from its base belief estimates.
• Figure 5: In the Nutrition domain, deceptive prompting generally raises belief misalignment, as models are incentivized to exaggerate or invent health benefits. However, instruction-tuned large models (e.g., Llama-3.1-70B) show resistance, likely due to strong instruction-tuning and safety behaviors that resist overt deception. Truthful prompting often lowers belief misalignment (e.g., gpt-4o-mini, gemma-2-27b-it) but can lead to higher belief misalignment if the model overcompensates with excessive disclaimers (e.g., Llama-3.1-70B-Instruct). Utilitarian prompting is mostly neutral-to-reducing, with exceptions (e.g., gemma-2-27b-it), possibly because maximizing utility in health contexts still encourages optimistic but slightly fact-skewed recommendations.