Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Ensemble-Directed Offline Reinforcement Learning

Desik Rengarajan, Nitin Ragothaman, Dileep Kalathil, Srinivas Shakkottai

TL;DR

The FEDORA codebase is developed, which distills the collective wisdom of the clients using an ensemble learning approach, and it is shown that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets.

Abstract

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Naïvely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.

Federated Ensemble-Directed Offline Reinforcement Learning

TL;DR

The FEDORA codebase is developed, which distills the collective wisdom of the clients using an ensemble learning approach, and it is shown that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets.

Abstract

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Naïvely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.
Paper Structure (33 sections, 13 equations, 16 figures, 2 algorithms)

This paper contains 33 sections, 13 equations, 16 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Federated Offline Reinforcement Learning
  5. Issues with Federated Offline RL
  6. FEDORA Design Approach
  7. Ensemble-Directed Learning over Client Policies
  8. Federated Optimism for Critic Training
  9. Proximal Policy Update for Heterogeneous Data
  10. Decaying the Influence of Local Data
  11. Experimental Evaluation
  12. Experiments on Simulated Environments
  13. Real-World Experiments on TurtleBot
  14. Conclusion
  15. Ethics Statement and Societal Impacts
  16. ...and 18 more sections

Figures (16)

  • Figure 1: Performance comparison of federated and centralized offline RL algorithms.
  • Figure 2: Evaluation of algorithms on different MuJoCo environments.
  • Figure 3: Comparison of FEDORA and centralized training with heterogeneous data.
  • Figure 4: Effect of varying the number of (a) local gradient steps, (b) participating clients in each round, and (c) expert clients in FEDORA.
  • Figure 5: Evaluation of FEDORA and other federated baselines for a mobile robot navigation task in the presence of an obstacle.
  • ...and 11 more figures