Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning

TL;DR

This work tackles the challenge of teaching language models to engage in productive, human-like discussions in partially observable multi-agent environments without human demonstrations. By decomposing communication into listening and speaking and using dense, environment-grounded rewards (notably imposter prediction) to guide dialogue, the authors train RWKV-based crewmates in an Among Us–style game. The approach leverages RL, auxiliary listening and speaking losses, and iterative self-play against adaptive imposters, achieving substantial improvements in win rates and producing emergent behaviors like direct accusations and evidence sharing. The results generalize across varied environments, and the work highlights both gains and failure modes, offering a pathway to scalable, self-improving multi-agent communication with language models in novel tasks.

Abstract

Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking. Our key idea is to leverage the agent's goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model's listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model's speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us, where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/

Figures (5)

• Figure 1: Examples of the gameplay and discussion phases of Among Us. During gameplay, all agents navigate a 2D grid environment (a 1-by-2 grid in this case, with two rooms at (0,0) and (1,0)), where agents can see everything in their same room. Here, the red, green, and yellow agents are in room (1,0), and the purple and blue agents are in room (0,0). Crewmates can perform tasks (indicated by the stars -- in this example there are 3 tasks), while imposters kill crewmates. Here, the orange and green agents are working on tasks. Agents can also report dead bodies, as the purple agent is currently doing, which initiates the discussion phase. During discussion phases, agents leverage large language models to generate free-form messages guided by our framework encouraging effective speaking and listening within the crewmates and finally vote out a suspected imposter. The example discussion shown on the right is based on a generated discussion from our trained models.
• Figure 2: Diagram of the embodied gameplay loop. The environment starts by sending observations to all agents simultaneously and collects tokenized actions from a set of valid actions at each timestep.
• Figure 3: Win rates for crewmates trained with different algorithms over the "base" environment: $2 \times 2$ grid of rooms, 4 tasks per crewmate, and 5 players. Error bars represent the maximum and minimum expected win rates across the three independently trained runs with different seeds.
• Figure 4: Win rates for crewmates trained with different algorithms over different configurations of the environment, modifying the environment shape, tasks, and number of players.
• Figure 5: Exploitability curves for policies over self-play iterations, evaluated on the base environment. The orange line indicates the expected win rate against an adversarially trained imposter. The black line indicates the expected win rate of crewmates who are specifically optimized against this iteration's imposters. Note that iteration 0 refers to the base models, while iteration 1 refers to the crewmate policy from the Cooperative Training section. Shaded regions represent the maximum and minimum win rates across the three independently trained runs with different seeds.