Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach

TL;DR

Policy transfer between related robotic tasks often suffers covariance shift when demonstrations are offline. This paper introduces an active LfD approach that extends the EARLY framework to policy transfer, enabling online demonstrations to be queried during transfer with a budget $N_d$ and guided by a trajectory-uncertainty threshold. The method situates itself on top of AWAC, integrating online demonstrations and adaptive querying to achieve higher success rates and better sample efficiency than AWAC, BC, and EARLY across eight transfer scenarios and preliminary sim-to-real tests on a Franka manipulator. The results demonstrate significant data-efficient transfer capability across diverse environments and embodiments, with potential impact for real-world robot adaptation where annotation costs are high.

Abstract

Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.

Figures (4)

• Figure 1: Overview of our method.
• Figure 2: Overview of all the source and target tasks.
• Figure 3: Results of simulation experiments.
• Figure 4: The scenarios for real-world experiments.