A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning
Paul Daoudi, Christophe Prieur, Bogdan Robu, Merwan Barlier, Ludovic Dos Santos
TL;DR
The paper tackles the problem of transferring policies between source and target environments with differing dynamics under a few-shot data regime. It introduces FOOD, a conservative objective that penalizes the source-trained policy using a trajectory-based divergence estimated via Imitation Learning, and demonstrates improved performance across diverse off-dynamics benchmarks with limited target data. The key contributions are a theoretical reduction bound linking target and source performance via trajectory divergences, a practical FOOD algorithm that combines online RL with imitation-learning surrogates, and extensive experiments showing robustness and superiority over baselines in most off-dynamics scenarios. This work advances data-efficient, safe transfer in reinforcement learning and lays groundwork for integrating additional IL techniques to further stabilize learning in real-world, dynamic variations.
Abstract
Off-dynamics Reinforcement Learning (ODRL) seeks to transfer a policy from a source environment to a target environment characterized by distinct yet similar dynamics. In this context, traditional RL agents depend excessively on the dynamics of the source environment, resulting in the discovery of policies that excel in this environment but fail to provide reasonable performance in the target one. In the few-shot framework, a limited number of transitions from the target environment are introduced to facilitate a more effective transfer. Addressing this challenge, we propose an innovative approach inspired by recent advancements in Imitation Learning and conservative RL algorithms. The proposed method introduces a penalty to regulate the trajectories generated by the source-trained policy. We evaluate our method across various environments representing diverse off-dynamics conditions, where access to the target environment is extremely limited. These experiments include high-dimensional systems relevant to real-world applications. Across most tested scenarios, our proposed method demonstrates performance improvements compared to existing baselines.