Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, Shimon Whiteson

TL;DR

This study challenges the prevailing view that centralized value-function methods are essential for strong cooperative MARL performance. By adapting PPO into an independent-learning framework (IPPO) with per-agent clipping and a shared local critic, the authors demonstrate competitive or superior results to state-of-the-art CTDE approaches on SMAC, including on several hard maps. Ablation studies reveal that policy clipping—rather than a reduced learning rate or centralized state information—drives many of IPPO's advantages, though the benefit of central state information remains inconsistent. The work suggests a shift in MARL research focus toward robust independent learning methods and questions the necessity of relative overgeneralization constraints in practical settings like SMAC. Overall, IPPO emerges as a scalable, robust alternative that expands the toolbox for cooperative multi-agent learning.

Abstract

Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

TL;DR

This study challenges the prevailing view that centralized value-function methods are essential for strong cooperative MARL performance. By adapting PPO into an independent-learning framework (IPPO) with per-agent clipping and a shared local critic, the authors demonstrate competitive or superior results to state-of-the-art CTDE approaches on SMAC, including on several hard maps. Ablation studies reveal that policy clipping—rather than a reduced learning rate or centralized state information—drives many of IPPO's advantages, though the benefit of central state information remains inconsistent. The work suggests a shift in MARL research focus toward robust independent learning methods and questions the necessity of relative overgeneralization constraints in practical settings like SMAC. Overall, IPPO emerges as a scalable, robust alternative that expands the toolbox for cooperative multi-agent learning.

Abstract

Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.

Paper Structure

This paper contains 15 sections, 6 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Dec-POMDPs.
  5. Centralised learning with decentralised execution.
  6. Trust Region Policy Optimization
  7. Independent PPO
  8. Empirical Results
  9. IPPO Performance
  10. Role of Centralised Value Functions
  11. Ablation Studies
  12. Discussion
  13. Conclusion
  14. Acknowledgments and Disclosure of Funding
  15. Appendix

Figures (6)

  • Figure 1: Results on select SMAC maps, comparing IPPO to QMIX and IQL. Uncertainty regions depict $[0.25,0.75]$ confidence intervals of the median test win rate.
  • Figure 2: Median win test rates on select SMAC maps at 10M steps, comparing IPPO to MAPPO results reported by anonymous_benchmarking_2020.
  • Figure 3: Results on select SMAC maps comparing IPPO and MAPPO.
  • Figure 4: Ablation study for IPPO with different combinations of policy and value clipping
  • Figure 5: Results on select SMAC maps comparing IPPO and IPPO-C with QMIX, VDN and IQL
  • ...and 1 more figures