Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis

TL;DR

The theoretical global convergence bound of AFedPG is analyzed, the advantage of the proposed algorithm in terms of both the sample complexity and time complexity is characterized, and the improved performance of AFedPG is empirically verified in four widely used MuJoCo environments.

Abstract

To improve the efficiency of reinforcement learning (RL), we propose a novel asynchronous federated reinforcement learning (FedRL) framework termed AFedPG, which constructs a global model through collaboration among $N$ agents using policy gradient (PG) updates. To address the challenge of lagged policies in asynchronous settings, we design a delay-adaptive lookahead technique \textit{specifically for FedRL} that can effectively handle heterogeneous arrival times of policy gradients. We analyze the theoretical global convergence bound of AFedPG, and characterize the advantage of the proposed algorithm in terms of both the sample complexity and time complexity. Specifically, our AFedPG method achieves $O(\frac{ε^{-2.5}}{N})$ sample complexity for global convergence at each agent on average. Compared to the single agent setting with $O(ε^{-2.5})$ sample complexity, it enjoys a linear speedup with respect to the number of agents. Moreover, compared to synchronous FedPG, AFedPG improves the time complexity from $O(\frac{t_{\max}}{N})$ to $O({\sum_{i=1}^{N} \frac{1}{t_{i}}})^{-1}$, where $t_{i}$ denotes the time consumption in each iteration at agent $i$, and $t_{\max}$ is the largest one. The latter complexity $O({\sum_{i=1}^{N} \frac{1}{t_{i}}})^{-1}$ is always smaller than the former one, and this improvement becomes significant in large-scale federated settings with heterogeneous computing powers ($t_{\max}\gg t_{\min}$). Finally, we empirically verify the improved performance of AFedPG in four widely used MuJoCo environments with varying numbers of agents. We also demonstrate the advantages of AFedPG in various computing heterogeneity scenarios.

Figures (8)

• Figure 1: An illustration of the asynchronous federated policy gradient updates. Each agent has a local copy of the environment, and agents may collect data according to different local policies. At each iteration, the agent in the yellow color finishes the local process and then communicates with the server, while the other agents keep sampling and computing local gradients in parallel. In the $k$-th global iteration, $\delta_{k} \in\mathbb{N}$ is the delay, $\widetilde{\tau}_{k-\delta_{k} }$ is the sample collected according to the policy $\pi_{\widetilde{\theta}_{k-\delta_{k} }}$, and $d_{k-\delta_{k}}$ is the updating direction calculated from the sample $\widetilde{\tau}_{k-\delta_{k}}$.
• Figure 2: Visualization of the four MuJoCo tasks considered in this paper for experiments.
• Figure 3: Reward performances of AFedPG ($N=2,4,8$) and PG ($N=1$) on various MuJoCo environments, where $N$ is the number of federated agents. The solid lines are averaged results over $10$ runs with random seeds from $0$ to $9$. The shadowed areas are confidence intervals with $95\%$ confidence level.
• Figure 4: Global time of AFedPG and FedPG with certain numbers of collected samples on various MuJoCo environments, where $N$ is the number of federated agents. The solid lines are averaged results over $10$ runs. The shadowed areas are confidence intervals with $95\%$ confidence level.
• Figure 5: Comparison of time consumptions between synchronous and asynchronous approaches. The circled numbers denote the indices of global steps.

Theorems & Definitions (14)

• Definition 4.1
• Definition 4.2
• Theorem 5.1
• Theorem 5.2
• Lemma B.4
• Lemma B.5
• proof
• Lemma B.6
• Lemma B.7
• Lemma B.8