Federated reinforcement learning for robot motion planning with zero-shot generalization

TL;DR

A federated reinforcement learning framework that enables collaborative learning of multiple learners and a central server without sharing their raw data and leverages on the derived zero-shot generalization guarantees on arrival time and safety.

Abstract

This paper considers the problem of learning a control policy for robot motion planning with zero-shot generalization, i.e., no data collection and policy adaptation is needed when the learned policy is deployed in new environments. We develop a federated reinforcement learning framework that enables collaborative learning of multiple learners and a central server, i.e., the Cloud, without sharing their raw data. In each iteration, each learner uploads its local control policy and the corresponding estimated normalized arrival time to the Cloud, which then computes the global optimum among the learners and broadcasts the optimal policy to the learners. Each learner then selects between its local control policy and that from the Cloud for next iteration. The proposed framework leverages on the derived zero-shot generalization guarantees on arrival time and safety. Theoretical guarantees on almost-sure convergence, almost consensus, Pareto improvement and optimality gap are also provided. Monte Carlo simulation is conducted to evaluate the proposed framework.

Figures (13)

• Figure 1: Implementation FedGen for learner $i$ in iteration $k$
• Figure 2: Parameter update logic at each iteration
• Figure 3: A sample environment in PyBullet
• Figure 4: Generalized performances to unseen environments
• Figure 5: Comparison between initial policy, locally converged policy and globally converged policy

Theorems & Definitions (11)

• Theorem 3.1
• Lemma 3.5
• Theorem 3.6
• Theorem 3.7
• Theorem 4.1
• Lemma 4.2
• Lemma 4.3
• Lemma 4.4
• Lemma 4.5
• Lemma 4.6