Verification Required: The Impact of Information Credibility on AI Persuasion

TL;DR

This work tackles how information credibility shapes AI persuasion in high-stakes settings by introducing MixTalk, a strategic communication game where a sender blends verifiable and unverifiable claims and a receiver allocates a verification budget to infer private state. The study evaluates five state-of-the-art LLM agents in 30 deployment-inspired environments (Recruitment, Insurance Claims, Used Cars) and proposes Tournament Oracle Policy Distillation (TOPD), an offline, prompt-based method that distills tournament-derived insights into inference-time guidance. Key contributions include a scalable mixed-credibility framework that captures unraveling, selective disclosure, and verification dynamics, along with rich behavioral metrics and empirical game analysis (including $\alpha$-Rank) that reveal non-transitive interactions and role specialization. The findings show that TOPD consistently improves receiver robustness and reduces oracle regret, offering a practical route to safer, more reliable AI-mediated decision-making without fine-tuning. Overall, the work provides a principled, scalable platform for studying and mitigating manipulation risks in LLM-to-LLM communication through credibility-aware prompts and verification-aware strategies.

Abstract

Agents powered by large language models (LLMs) are increasingly deployed in settings where communication shapes high-stakes decisions, making a principled understanding of strategic communication essential. Prior work largely studies either unverifiable cheap-talk or fully verifiable disclosure, failing to capture realistic domains in which information has probabilistic credibility. We introduce MixTalk, a strategic communication game for LLM-to-LLM interaction that models information credibility. In MixTalk, a sender agent strategically combines verifiable and unverifiable claims to communicate private information, while a receiver agent allocates a limited budget to costly verification and infers the underlying state from prior beliefs, claims, and verification outcomes. We evaluate state-of-the-art LLM agents in large-scale tournaments across three realistic deployment settings, revealing their strengths and limitations in reasoning about information credibility and the explicit behavior that shapes these interactions. Finally, we propose Tournament Oracle Policy Distillation (TOPD), an offline method that distills tournament oracle policy from interaction logs and deploys it in-context at inference time. Our results show that TOPD significantly improves receiver robustness to persuasion.

Figures (5)

• Figure 1: MixTalk leaderboard for strategic communication skills.
• Figure 2: The sender optimizes the submitted claim by strategically selecting which information to disclose. Here, the sender reports the lab test and reinforces the diagnosis, but omits risk factors to pursue a stronger symptom narrative. The claim screening agent verifies a subset of the disclosed information and expresses pessimism about the missing risk factors. However, due to its belief, the omission is incorrectly associated with severe complaints, leading to an overly favorable claim assessment.
• Figure 3: Tournament payoff matrices.
• Figure 4: Credibility counterfactual behaviors. Comparison of CheapTalk, MixTalk, and the Disclosure game.
• Figure A1: Payoff matrices. Includes combined tournament payoff matrices for senders (top) and receivers (bottom).