Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning

Alexander Tyurin, Andrei Spiridonov, Varvara Rudenko

Abstract

We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achieved remarkable empirical success, their distributed counterparts remain less explored, particularly in the presence of heterogeneous asynchronous computations and communication bottlenecks. We introduce two new algorithms, Rennala NIGT and Malenia NIGT, which implement asynchronous policy gradient aggregation and achieve state-of-the-art efficiency. In the homogeneous setting, Rennala NIGT provably improves the total computational and communication complexity while supporting the AllReduce operation. In the heterogeneous setting, Malenia NIGT simultaneously handles asynchronous computations and heterogeneous environments with strictly better theoretical guarantees. Our results are further corroborated by experiments, showing that our methods significantly outperform prior approaches.

Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning

Abstract

We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achieved remarkable empirical success, their distributed counterparts remain less explored, particularly in the presence of heterogeneous asynchronous computations and communication bottlenecks. We introduce two new algorithms, Rennala NIGT and Malenia NIGT, which implement asynchronous policy gradient aggregation and achieve state-of-the-art efficiency. In the homogeneous setting, Rennala NIGT provably improves the total computational and communication complexity while supporting the AllReduce operation. In the heterogeneous setting, Malenia NIGT simultaneously handles asynchronous computations and heterogeneous environments with strictly better theoretical guarantees. Our results are further corroborated by experiments, showing that our methods significantly outperform prior approaches.

Paper Structure

This paper contains 31 sections, 28 theorems, 92 equations, 10 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Problem Formulation and Preliminaries
  5. Assumptions
  6. Homogeneous and Heterogeneous Setups
  7. Computation and Communication Times
  8. New Methods: Rennala NIGT and Malenia NIGT
  9. Time Complexity in the Homogeneous Setup
  10. Time complexity with communication times
  11. Time Complexities in the Heterogeneous Setup
  12. Summary of Experiments
  13. Universal Computation Model
  14. Global Convergence
  15. Notations
  16. ...and 16 more sections

Key Result

Proposition 2.3

Let Assumptions Assumption_1 and Assumption_2 hold. Then,

Figures (10)

  • Figure 1: Experiments on Humanoid-v4 with increasing heterogeneity of times (from left to right).
  • Figure 2: Experiments on Humanoid-v4 with increasing heterogeneity of times (from left to right).
  • Figure 3: Experiments on MuJoCo tasks with $h_i = \sqrt{i}$, $\kappa_i = \sqrt{i} \times d^{1/4}$.
  • Figure 4: Experiments on MuJoCo tasks with $h_i = 1$, $\kappa_i = 0$ and $n = 100$ (similar rates since times are equal).
  • Figure 5: Experiments on MuJoCo tasks with $h_i = \sqrt{i}$, $\kappa_i = 0$ and $n = 100$.
  • ...and 5 more figures

Theorems & Definitions (42)

  • Proposition 2.3: e.g. Zhang2020bMasiha2022Yuan2022b
  • Proposition 2.4
  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 5.1
  • Theorem 7.2
  • Theorem 7.3
  • Theorem A.1
  • ...and 32 more