Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects

TL;DR

MARL suffers from high sample complexity due to non-stationarity and coordination among multiple agents. This paper surveys model-based MARL, offering a taxonomy and theoretical insights into sample efficiency across centralized and decentralized settings. It highlights that learned environment dynamics and planning can improve data efficiency in two-agent and general-sum games, while noting remaining challenges such as model inaccuracy and credit assignment. The work points to promising directions—scalability, opponent modeling, and learned communication—that may enable scalable, robust MARL in real-world multi-agent systems.

Abstract

Significant advances have recently been achieved in Multi-Agent Reinforcement Learning (MARL) which tackles sequential decision-making problems involving multiple participants. However, MARL requires a tremendous number of samples for effective training. On the other hand, model-based methods have been shown to achieve provable advantages of sample efficiency. However, the attempts of model-based methods to MARL have just started very recently. This paper presents a review of the existing research on model-based MARL, including theoretical analyses, algorithms, and applications, and analyzes the advantages and potential of model-based MARL. Specifically, we provide a detailed taxonomy of the algorithms and point out the pros and cons for each algorithm according to the challenges inherent to multi-agent scenarios. We also outline promising directions for future development of this field.