Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox

TL;DR

This paper surveys distributed deep reinforcement learning (DDRL) from single-agent to multi-player multi-agent settings, clarifying how coordination, data throughput, and agent cooperation shape algorithm design. It introduces a three-part taxonomy (coordinators, agents cooperation, and player evolution) and reviews a spectrum of toolboxes (Ray RLlib, Acme, SeedRL, MALib, Tianshou, TorchBeast) while highlighting their strengths and limitations. A key contribution is the presentation of M2RL, a multi-player multi-agent toolbox with per-player learners, actors, experience buffers, and a players manager, demonstrated on a complex Wargame to validate MP-MA training. The work provides a practical guide for researchers choosing DDRL methods and tools and discusses challenges such as large-scale testing, big-model training, and the integration of self-play/population-play paradigms for robust, real-world applications.

Abstract

With the breakthrough of AlphaGo, deep reinforcement learning becomes a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning hard to be practical in a wide range of areas. Plenty of methods have been developed for sample efficient deep reinforcement learning, such as environment modeling, experience transfer, and distributed modifications, amongst which, distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming, and intelligent transportation. In this paper, we conclude the state of this exciting field, by comparing the classical distributed deep reinforcement learning methods, and studying important components to achieve efficient distributed learning, covering single player single agent distributed deep reinforcement learning to the most complex multiple players multiple agents distributed deep reinforcement learning. Furthermore, we review recently released toolboxes that help to realize distributed deep reinforcement learning without many modifications of their non-distributed versions. By analyzing their strengths and weaknesses, a multi-player multi-agent distributed deep reinforcement learning toolbox is developed and released, which is further validated on Wargame, a complex environment, showing usability of the proposed toolbox for multiple players and multiple agents distributed deep reinforcement learning under complex games. Finally, we try to point out challenges and future trends, hoping this brief review can provide a guide or a spark for researchers who are interested in distributed deep reinforcement learning.

Figures (17)

• Figure 1: Basic framework of DDRL.
• Figure 2: Single player single agent DDRL to multiple players multiple agents DDRL.
• Figure 3: The taxonomy of distributed deep reinforcement learning.
• Figure 4: Basic framework of Gorila.
• Figure 5: Basic framework of A3C.