SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning

TL;DR

This paper addresses credit assignment and scalability challenges in large-scale cooperative MARL. It proposes SrSv, a Transformer-based paradigm that combines sequential action rollouts with sequential value estimation, integrating per-agent value estimation into an attention-based sequential model. SrSv demonstrates faster training and robust convergence on SMAC and MA-MuJoCo benchmarks and scales to 1024 agents in a DubinsCar setting, outperforming baselines MAPPO, A2PO, and MAT. These results suggest that explicit per-agent value estimation aligned with sequential decision making enables scalable coordination in real-world multi-agent systems, with potential implications for safe RL and industrial deployments.

Abstract

Although multi-agent reinforcement learning (MARL) has shown its success across diverse domains, extending its application to large-scale real-world systems still faces significant challenges. Primarily, the high complexity of real-world environments exacerbates the credit assignment problem, substantially reducing training efficiency. Moreover, the variability of agent populations in large-scale scenarios necessitates scalable decision-making mechanisms. To address these challenges, we propose a novel framework: Sequential rollout with Sequential value estimation (SrSv). This framework aims to capture agent interdependence and provide a scalable solution for cooperative MARL. Specifically, SrSv leverages the autoregressive property of the Transformer model to handle varying populations through sequential action rollout. Furthermore, to capture the interdependence of policy distributions and value functions among multiple agents, we introduce an innovative sequential value estimation methodology and integrates the value approximation into an attention-based sequential model. We evaluate SrSv on three benchmarks: Multi-Agent MuJoCo, StarCraft Multi-Agent Challenge, and DubinsCars. Experimental results demonstrate that SrSv significantly outperforms baseline methods in terms of training efficiency without compromising convergence performance. Moreover, when implemented in a large-scale DubinsCar system with 1,024 agents, our framework surpasses existing benchmarks, highlighting the excellent scalability of SrSv.

Figures (7)

• Figure 1: The modeling of agent interdependence for sequential decision-making.
• Figure 2: The encoder-decoder architecture of SrSv.
• Figure 3: Demonstrations of the multi-agent benchmarks: MMM3 in SMAC, Half-Cheetah in MA-MuJoCo and DubinsCar.
• Figure 4: Performance comparisons on cooperative MARL benchmarks among SrSv and other baselines.
• Figure 5: Dead ratio metric comparisons on SMAC benchmark between SrSv and MAT.