Sotopia-RL: Reward Design for Social Intelligence

TL;DR

Sotopia-RL tackles the challenge of training socially intelligent LLMs by designing utterance-level, multi-dimensional rewards rather than relying solely on episode-level signals. The method offline-labels turn-level contributions using LLM rubrics and then online-trains a turn-level reward model to guide policy optimization via GRPO, starting from behavior cloning on a 7B-instruction-tuned base. It achieves state-of-the-art social goal completion on the Sotopia benchmark (7.17 on Sotopia-hard and 8.31 on Sotopia-full) and demonstrates that both utterance-level credit assignment and multi-dimensional reward aggregation are essential for stable, robust RL in social contexts. The results generalize across partner/evaluator models and align with human judgments, while analyses address safety and diversity considerations and point toward future extensions to personalized and multi-agent settings.

Abstract

Social intelligence has become a critical capability for large language models (LLMs), enabling them to engage effectively in real-world social tasks such as collaboration and negotiation. Reinforcement learning (RL) is a natural fit for training socially intelligent agents because it allows models to learn sophisticated strategies directly through social interactions without requiring human annotations. However, there are two unique parts about social intelligence tasks: (1) the quality of individual utterances in social interactions is not strictly related to final success; (2) social interactions require multi-dimensional rubrics for success. Therefore, we argue that it is necessary to design rewards for building utterance-level multi-dimensional reward models to facilitate RL training for social intelligence tasks. To address these challenges, we propose Sotopia-RL, a novel framework that refines coarse episode-level feedback into utterance-level, multi-dimensional rewards. Utterance-level credit assignment attributes outcomes to individual utterances, while multi-dimensional rewards capture the full richness of social interactions and reduce reward hacking. Experiments in Sotopia, an open-ended social learning environment, demonstrate that Sotopia-RL achieves state-of-the-art social goal completion scores (7.17 on Sotopia-hard and 8.31 on Sotopia-full), significantly outperforming existing approaches. Ablation studies confirm the necessity of both utterance-level credit assignment and multi-dimensional reward design for RL training.

Figures (18)

• Figure 1: Uniqueness of social intelligence tasks.(Left) Accommodation, persuasion, collaboration, and negotiation represent four core types of social intelligence tasks. Our method achieves consistent improvements across all tasks compared with Sotopia-$\pi$, the previous state-of-the-art on Sotopia. (Right) Two unique features of social interactions: (1) utterances are not always directly tied to outcomes—e.g., people may lie or mislead in negotiations; (2) interactions are inherently multi-dimensional—e.g., relationship building can play a crucial role in achieving task success.
• Figure 2: An example of a social task in the Sotopia environment. Tom is agent A, and Oliver is agent B. Each agent has a unique goal that is hidden from the other. "9 / 10" indicates a single-dimensional episode-level reward provided by LLMs to describe its goal completion status.
• Figure 3: Overview of the Sotopia-RL pipeline. Stage 1—Offline data preparation: generate GPT self-play dialogues and assign utterance-level rewards via offline inference with full-episode context. Stage 2—RL training: (2.1) SFT initializes the policy and distills an utterance-level reward model (RM) from offline labels; (2.2) Online RL method continues self-play using rewards from the online RM, which conditions only on dialogue history up to the current turn.
• Figure 4: Overview of social reward design. To better describe and model the quality of an utterance in social interactions, we expand the episode-level reward ("9/10" mentioned above) from two axes: (1) expanding from episode-level into utterance-level; (2) expanding from single-dimension to multi-dimensions, expanding from goal completion (Goal) to relationship maintaining (Rel) and knowledge seeking (Kno). It allows us to have denser reward signals for RL training.
• Figure 5: Evaluation results with different LLM-based evaluators. The consistent improvement on evaluators indicates no reward hacking. Full results in Appendix §\ref{['ablation_res']}.