MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs

TL;DR

MARSHAL tackles the challenge of generalizable multi-agent reasoning for LLMs by introducing turn-level credit assignment and agent-specific normalization within a self-play framework built on Group-Relative Policy Optimization. It employs a diverse set of cooperative and competitive games to instill robust strategic reasoning and demonstrates transfer to downstream multi-agent systems (MAD and AutoGen) with notable gains in AIME and GPQA benchmarks. Comprehensive ablations and reasoning-pattern analyses corroborate that both the turn-level estimator and role-aware normalization are critical for stability and generalization in multi-turn, multi-agent settings. Overall, the work provides strong evidence that end-to-end RL with self-play can cultivate generalizable, multi-agent reasoning capabilities in LLMs applicable to MAS tasks and reasoning benchmarks.

Abstract

Developing large language models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agent Reasoning through Self-play witH strAtegic LLMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agent trained from Qwen3-4B develops strong strategic abilities that generalize to held-out games with up to 28.7% performance improvements. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant performance gains of up to 10.0% on AIME, 6.6% on GPQA-Diamond, and 3.5% on average across all benchmarks. These results establish end-to-end RL training with self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs.

Figures (6)

• Figure 1: Evaluation of MARSHAL and two baselines on strategic games and reasoning benchmarks. MARSHAL incentivizes multi-agent reasoning ability via self-play in strategic games and generalizes to improvements of multi-agent systems like MAD and AutoGen on math and QA benchmarks.
• Figure 2: Overview of MARSHAL. Left column: generating player trajectories through self-play in strategic games. Middle column: naive advantage estimation by GRPO. Right column: advantage estimation by MARSHAL for accurate credit assignment in multi-turn, multi-agent setting.
• Figure 3: Average normalized game returns. Specialist agents not only master their training domains but also generalize effectively to their more complex, held-out counterparts (e.g., from Tic-Tac-Toe to Connect Four). The generalist model achieves consistently high performance across the entire suite of games, establishing it as the most robust and versatile agent.
• Figure 4: Eval curves of the Tic-Tac-Toe specialist in Tic-Tac-Toe and Mini Hanabi.
• Figure 5: Percentage of different failure modes in GPQA-Dimond. Through MARSHAL training, the generalist agent significantly reduces the occurrence of Inter-Agent Misalignment.