Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

Zhihong Liu, Xin Xu, Peng Qiao, Dongsheng Li

TL;DR

This article performs a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references.

Abstract

Deep reinforcement learning has led to dramatic breakthroughs in the field of artificial intelligence for the past few years. As the amount of rollout experience data and the size of neural networks for deep reinforcement learning have grown continuously, handling the training process and reducing the time consumption using parallel and distributed computing is becoming an urgent and essential desire. In this paper, we perform a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references. In particular, a taxonomy of literature is provided, along with a discussion of emerging topics and open issues. This incorporates learning system architectures, simulation parallelism, computing parallelism, distributed synchronization mechanisms, and deep evolutionary reinforcement learning. Further, we compare 16 current open-source libraries and platforms with criteria of facilitating rapid development. Finally, we extrapolate future directions that deserve further research.

Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

TL;DR

This article performs a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references.

Abstract

Deep reinforcement learning has led to dramatic breakthroughs in the field of artificial intelligence for the past few years. As the amount of rollout experience data and the size of neural networks for deep reinforcement learning have grown continuously, handling the training process and reducing the time consumption using parallel and distributed computing is becoming an urgent and essential desire. In this paper, we perform a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references. In particular, a taxonomy of literature is provided, along with a discussion of emerging topics and open issues. This incorporates learning system architectures, simulation parallelism, computing parallelism, distributed synchronization mechanisms, and deep evolutionary reinforcement learning. Further, we compare 16 current open-source libraries and platforms with criteria of facilitating rapid development. Finally, we extrapolate future directions that deserve further research.

Paper Structure

This paper contains 32 sections, 3 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related surveys
  3. Structure of the survey
  4. Background
  5. Preliminaries of deep reinforcement learning
  6. Distributed training for deep learning
  7. Challenges of parallel and distributed training for DRL
  8. System Architectures for Distributed DRL
  9. Centralized architecture
  10. Decentralized architecture
  11. Discussion
  12. Simulation Parallelism
  13. Distributed simulation based on CPU clusters
  14. Batch simulation based on GPUs/TPUs
  15. Discussion
  16. ...and 17 more sections

Figures (7)

  • Figure 1: The structure of the survey.
  • Figure 2: The semantics of deep reinforcement learning.
  • Figure 3: Comparison of centralized and decentralized architectures for parallel and distributed training in DRL.
  • Figure 4: Architectures of current parallel and distributed DRL methods.
  • Figure 5: Comparison of two simulation parallelism pipelines. (a) The intermediate data needs to be copied from CPUs to GPUs back and forth during training in CPU simulation parallelism pipeline. (b) GPU simulation of zero-copy enables directly accessing to simulation results in the GPU buffers and keeps all of the computations on the GPU. Makoviychuk2021
  • ...and 2 more figures