An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning
Christopher Amato
TL;DR
This work surveys centralized training for decentralized execution in cooperative multi-agent reinforcement learning by formalizing Dec-POMDPs and contrasting training paradigms. It delineates two major families of CTDE methods: value-function factorization (VDN, QMIX, QTRAN, QPLEX, and variants) and centralized-critic actor-critic methods (MADDPG, COMA, MAPPO/IPPO), clarifying their theoretical underpinnings, architectures, and tradeoffs. The discussion emphasizes the importance of the IGM principle, how state or history is used in factorization, and the pitfalls of state-based critics in partial observability. It also contrasts CTDE with decentralized training and outlines practical considerations such as scalability, nonstationarity, and the role of parameter sharing. Overall, the text offers a compact, guidance-oriented synthesis of core CTDE concepts and methods for cooperative MARL.
Abstract
Multi-agent reinforcement learning (MARL) has exploded in popularity in recent years. Many approaches have been developed but they can be divided into three main types: centralized training and execution (CTE), centralized training for decentralized execution (CTDE), and Decentralized training and execution (DTE). CTDE methods are the most common as they can use centralized information during training but execute in a decentralized manner -- using only information available to that agent during execution. CTDE is the only paradigm that requires a separate training phase where any available information (e.g., other agent policies, underlying states) can be used. As a result, they can be more scalable than CTE methods, do not require communication during execution, and can often perform well. CTDE fits most naturally with the cooperative case, but can be potentially applied in competitive or mixed settings depending on what information is assumed to be observed. This text is an introduction to CTDE in cooperative MARL. It is meant to explain the setting, basic concepts, and common methods. It does not cover all work in CTDE MARL as the subarea is quite extensive. I have included work that I believe is important for understanding the main concepts in the subarea and apologize to those that I have omitted.