Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game

TL;DR

This work introduces MaKTO, a language-model training framework grounded in Wittgenstein-inspired language game theory that unifies language, action, and intent within a multi-agent Werewolf environment. By combining expert data collection, behavior cloning, and Kahneman-Tversky Optimization with a diverse model pool, MaKTO learns robust, stepwise decision policies without requiring paired language-decision data. The approach yields state-of-the-art performance in 9-player Werewolf settings, achieving a 61% average win rate against strong baselines and about 60% against expert humans, while remaining largely indistinguishable from human play in Turing-style tests. These results demonstrate the practicality of in-context, interaction-driven learning for complex strategic dialogue tasks and offer a scalable dataset for Werewolf gameplay with rich reasoning traces.

Abstract

Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make stratigic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory in Philosophical Investigations, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, we develop the Multi-agent Kahneman & Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 49% detectability in Turing-style blind tests.

Figures (7)

• Figure 1: Language Theory and AI Architecture: Traditional vs. Language Game Models.(a) and (b): Multi-staged framework that separates language and decisions. (c) and (d): our proposed framework inspired by Wittgenstein's language game theory, integrating language, actions, and intentions in a multi-agent game.
• Figure 2: The overall training process consists of (1) behavior cloning using instruction data ($\S$\ref{['sec:method_sft']}) and (2) multi-agent KTO($\S$\ref{['sec:method_kto']}). In multi-agent gameplay, we randomly assign roles to agents to create diverse interactions that optimize the target model. A stepwise selection process (Right) identifies desirable and unacceptable preference data using three methods: heuristic-based, staged voting-based, and verifier-based selection. The preference data is finally used for KTO optimization.
• Figure 3: Villager win rate matrix of the head-to-head competition: villager models (y-axis) vs. werewolf models (x-axis). Lower left: higher values show stronger villager performance; Upper right: lower values indicate stronger werewolf performance.
• Figure 4: Results of 260 random competitions in 9-player Seer-Witch-Guard game setting.
• Figure 5: Win rate of players in random competition. H1-14 stand for the win rate of human players.