MARO: Learning Stronger Reasoning from Social Interaction

TL;DR

This work addresses the absence of experiential social interaction in LLM training by introducing MARO, a framework that uses multi-agent social simulations to elicit dense, step-wise rewards from final outcomes. By decomposing sparse success signals, balancing training across roles, and evaluating per-behavior utility with a log-likelihood-based implicit reward, MARO improves social reasoning and generalizes to mathematical reasoning and instruction following. Experiments in MIRAGE murder-mystery environments show MARO achieving superior social capability across simple and complex settings, with stronger transfer effects in more challenging contexts. The findings suggest multi-agent social learning as a scalable route to broad, cross-domain reasoning enhancement for LLMs, with demonstrated architecture-agnostic applicability (e.g., Qwen and Llama-based models).

Abstract

Humans face countless scenarios that require reasoning and judgment in daily life. However, existing large language model training methods primarily allow models to learn from existing textual content or solve predetermined problems, lacking experience in real scenarios involving interaction, negotiation, and competition with others. To address this, this paper proposes Multi-Agent Reward Optimization (MARO), a method that enables large language models (LLMs) to acquire stronger reasoning abilities by learning and practicing in multi-agent social environments. Specifically, MARO first addresses the sparse learning signal problem by decomposing final success or failure outcomes into each specific behavior during the interaction process; second, it handles the uneven role distribution problem by balancing the training sample weights of different roles; finally, it addresses environmental instability issues by directly evaluating the utility of each behavior. Experimental results demonstrate that MARO not only achieves significant improvements in social reasoning capabilities, but also that the abilities acquired through social simulation learning can effectively transfer to other tasks such as mathematical reasoning and instruction following. This reveals the tremendous potential of multi-agent social learning in enhancing the general reasoning capabilities of LLMs.

Figures (10)

• Figure 1: Overview of the MARO workflow. First, agents interact in social scenarios where each agent pursues individual goals through communication and decision-making. Second, upon interaction completion, the system evaluates final outcomes to determine success or failure for each participating agent. Third, MARO decomposes these sparse final outcomes into dense step-wise rewards distributed across each agent's action trajectory, incorporating role-specific weights for balanced training. Fourth, the decomposed reward data is used to optimize LLMs through MARO's specialized loss function for enhanced social reasoning capabilities.
• Figure 2: Performance comparison across different model types in various scenarios. The radar charts show five evaluation metrics (Persona, Interaction, Victory, Investigation, Trust) across four conditions: (a) Simple-k: killer faction performance in simple scenarios, (b) Complex-k: killer faction performance in complex scenarios, (c) Simple-V: Victim faction performance in simple scenarios, and (d) Complex-V: Victim faction performance in complex scenarios. All models (Vanilla, SFT, MAKTO, MARO) play both killer and Victim roles in their respective configurations.
• Figure 3: Comprehensive performance improvement heatmap showing absolute percentage point changes compared to Vanilla across all benchmarks.
• Figure 4: Relative improvement ratios of different training methods compared to Vanilla on mathematical reasoning tasks.
• Figure 5: Performance comparison when the killer faction is enhanced (SFT-k, MARO-k). The killer role is controlled by enhanced models while the victim role is controlled by Vanilla.