SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning

TL;DR

The paper identifies critical deficiencies in the StarCraft Multi-Agent Challenge (SMAC) related to insufficient stochasticity and meaningless partial observability, which allow open-loop policies to achieve non-trivial win rates. To address this, the authors introduce SMACv2 with procedurally generated scenarios, random start positions, unit-type diversity, and true-range dynamics, plus an Extended Partial Observability (EPO) variant to enforce meaningful hidden information and implicit communication. Through extensive experiments with QMIX, MAPPO, QPLEX, and open-loop baselines, SMACv2 demonstrates substantially greater difficulty and reveals that strong performance in SMAC does not translate to SMACv2. The results underscore the importance of stochasticity and meaningful partial observability for evaluating closed-loop, decentralized policies, and advocate evaluating MARL methods across multiple benchmarks to avoid overfitting. Overall, SMACv2 provides a rigorous, extensible platform to drive advances toward generalizable and communicative cooperative MARL methods.

Abstract

The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become a popular testbed for centralised training with decentralised execution. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies. In particular, we show that an *open-loop* policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We show that these changes ensure the benchmark requires the use of *closed-loop* policies. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available at https://sites.google.com/view/smacv2.

Figures (14)

• Figure 1: Screenshots from SMACv2 showing agents battling the built-in AI.
• Figure 2: QMIX (left) and MAPPO (right) open-loop and closed-loop results.
• Figure 3: Comparison of the loss for different feature masks when regressing to a trained QMIX policy. The x-axis plots steps within an episode. 'Episodes Running' is the number of episodes that have not terminated by the given timestep.everything masks the entire observation and nothing masks no attributes. team_all masks all attributes of the team. last_action_only masks the last actions of all the allies in the state. The all_except_actions masks all the allied information except the last actions in the state. The mean is plotted for each mask and standard deviation across 3 seeds used as error bars. The low error rates for the everything mask imply that Q-values can be effectively inferred given only the timestep.
• Figure 4: Examples of the two different types of start positions, reflect and surround. Allied units are shown in blue and enemy units in dark red.
• Figure 5: Example EPO enemy sighting. Allied units that do not observe the enemy are shown in blue, those that do are shown in green and the enemy unit in dark red. Initially, an ally spots an enemy. Later (right), when the enemy is within all allied sight ranges, only the first ally to observe the enemy and the ally for which the draw was successful can see it.