ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind

TL;DR

ToMAP addresses the limitation of LLM persuaders lacking Theory of Mind by introducing two ToM modules—Counterclaim Predictor and Opponent Attitude Predictor—coupled with reinforcement learning to train an opponent-aware persuader. The 3B-parameter ToMAP demonstrates substantial gains over larger baselines across multiple persuadee models and corpora, driven by more complex reasoning and reduced repetition. RL enables ToMAP to leverage ToM insights for dynamic, long-turn conversations, yielding more diverse and effective arguments. The work highlights the practical potential of integrating theory of mind into persuasive language agents and provides reproducible code and safety considerations.

Abstract

Large language models (LLMs) have shown promising potential in persuasion, but existing works on training LLM persuaders are still preliminary. Notably, while humans are skilled in modeling their opponent's thoughts and opinions proactively and dynamically, current LLMs struggle with such Theory of Mind (ToM) reasoning, resulting in limited diversity and opponent awareness. To address this limitation, we introduce Theory of Mind Augmented Persuader (ToMAP), a novel approach for building more flexible persuader agents by incorporating two theory of mind modules that enhance the persuader's awareness and analysis of the opponent's mental state. Specifically, we begin by prompting the persuader to consider possible objections to the target central claim, and then use a text encoder paired with a trained MLP classifier to predict the opponent's current stance on these counterclaims. Our carefully designed reinforcement learning schema enables the persuader learns how to analyze opponent-related information and utilize it to generate more effective arguments. Experiments show that the ToMAP persuader, while containing only 3B parameters, outperforms much larger baselines, like GPT-4o, with a relative gain of 39.4% across multiple persuadee models and diverse corpora. Notably, ToMAP exhibits complex reasoning chains and reduced repetition during training, which leads to more diverse and effective arguments. The opponent-aware feature of ToMAP also makes it suitable for long conversations and enables it to employ more logical and opponent-aware strategies. These results underscore our method's effectiveness and highlight its potential for developing more persuasive language agents. Code is available at: https://github.com/ulab-uiuc/ToMAP.

Figures (6)

• Figure 1: Human thoughts about claims are inherently interconnected. While human persuaders can recognize this network of related propositions, LLMs often focus narrowly on the central claim alone and omit other claims.
• Figure 2: Overview of Theory of Mind Augmented Persuader (ToMAP). ToMAP utilizes two ToM modules, the counterclaim predictor and attitude predictor, to effectively model the opponent's mental state during the conversation. This design enables ToMAP to provide more diverse arguments and counter the persuadee's concerns in a more flexible and opponent-aware manner.
• Figure 3: LLM Persuadee and human experts have consistent judgement of persuasiveness.
• Figure 4: Plots for key metrics during RL training. Plots are smoothed for better readability.
• Figure 5: ToMAP shows steady gains in long conversations. The persuadee is QWen2.5-7B-Instruct.