One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence

TL;DR

OMAR presents a generalizable reinforcement learning framework that trains a single model to role-play all participants in multi-turn group conversations, enabling the emergence of long-horizon social intelligence through self-play. A hierarchical advantage estimation scheme stabilizes credit assignment across turns and tokens, enabling end-to-end PPO updates in a multi-agent, multi-turn setting. Empirical results in SOTOPIA and Werewolf demonstrate emergent behaviors such as empathy, persuasion, compromise, and collaboration, including improved fine-grained social metrics and higher team win rates under competitive dynamics. These findings suggest that rich social intelligence can arise from autonomous social learning in open-ended dialogue, while highlighting practical challenges like reward hacking and the need for scalable, robust evaluation and supervision strategies.

Abstract

This paper introduces OMAR: One Model, All Roles, a reinforcement learning framework that enables AI to develop social intelligence through multi-turn, multi-agent conversational self-play. Unlike traditional paradigms that rely on static, single-turn optimizations, OMAR allows a single model to role-play all participants in a conversation simultaneously, learning to achieve long-term goals and complex social norms directly from dynamic social interaction. To ensure training stability across long dialogues, we implement a hierarchical advantage estimation that calculates turn-level and token-level advantages. Evaluations in the SOTOPIA social environment and Werewolf strategy games show that our trained models develop fine-grained, emergent social intelligence, such as empathy, persuasion, and compromise seeking, demonstrating the effectiveness of learning collaboration even under competitive scenarios. While we identify practical challenges like reward hacking, our results show that rich social intelligence can emerge without human supervision. We hope this work incentivizes further research on AI social intelligence in group conversations.

Figures (6)

• Figure 1: Comparison of the standard GRPO framework (top) vs the proposed OMAR framework (bottom). While GRPO generates $n$ independent rollouts from a single prompt to calculate group-averaged advantages, OMAR repurposes this architecture for multi-agent, multi-turn conversations. In our framework, a single Policy Model role-plays $n$ distinct participants simultaneously by adding persona prompts to the shared initial prompt, with the batch size set equal to the number of active participants. Utterances from turn $t$ are aggregated to form the context for turn $t+1$ for each participant, creating a shared conversation history. Rewards are assigned at the end of the conversation based on environment-specific outcomes, such as consensus/abstention, satisfaction, win/loss, task completion, or turn limits, allowing the model to learn complex social intelligence through self-play.
• Figure 2: Hierarchical advantage estimation for multi-turn conversational RL. To mitigate high variance in reward propagation across long sequences, OMAR decouples advantage calculation into two stages. The turn-level stage (top) uses the final reward and values at the last token in each turn to compute turn-level advantages. In the token-level stage (bottom), these turn-level advantages are treated as pseudo-rewards and combined with token-level values to estimate final token-level advantages within that turn. All token-level advantages are then utilized to optimize the policy model. In the diagram, dark blue blocks represent turn-level components, while yellow blocks represent token-level elements. This framework utilizes GAE from PPO, as we no longer have $n$ independent rollouts for group-relative advantage estimation in GRPO.
• Figure 3: Evaluation results on the SOTOPIA Dataset. The top subfigure reports performance on SOTOPIA metrics that are directly optimized as training rewards. We omit results on secret and financial benefit metrics, as all models achieve near-zero scores with no meaningful variation. While these metrics reflect high-level social outcomes, they may not fully capture the fine-grained social behaviors that emerge in realistic multi-turn conversations. The bottom subfigure, therefore, presents zero-shot evaluation results on more fine-grained social intelligence metrics, where our model exhibits even larger performance gaps compared to baseline methods, showing the effectiveness of multi-turn, multi-agent RL.
• Figure 4: Example utterances illustrating social intelligence of our model trained under multi-turn, multi-agent reinforcement learning on SOTOPIA dataset.
• Figure 5: Example utterances illustrating social intelligence of our model trained under multi-turn, multi-agent reinforcement learning on Werewolf game dataset.