Relative Policy-Transition Optimization for Fast Policy Transfer
Jiawei Xu, Cheng Zhou, Yizheng Zhang, Baoxiang Wang, Lei Han
TL;DR
This work addresses policy transfer under model mismatch by introducing the Value Relativity Lemma, which decomposes the relativity gap between two MDPs into dynamics-induced and policy-induced components. Building on this, it proposes Relative Policy Optimization (RPO) for fast policy transfer and Relative Transition Optimization (RTO) for dynamics modeling, and combines them into Relative Policy-Transition Optimization (RPTO) for joint policy and transition learning across two environments. Theoretical bounds connect the relativity gap to tractable objectives, while practical SAC-based and weighted supervised losses enable efficient training on continuous-control tasks. Empirical results on MuJoCo tasks show that RPTO achieves faster, more robust transfer than strong baselines, with additional insights from a CartPole didactic example. The framework offers a principled, data-efficient approach to sim2real-like transfer and curriculum-style environment bridging, with potential extensions to differentiable simulators and meta-learning.
Abstract
We consider the problem of policy transfer between two Markov Decision Processes (MDPs). We introduce a lemma based on existing theoretical results in reinforcement learning to measure the relativity gap between two arbitrary MDPs, that is the difference between any two cumulative expected returns defined on different policies and environment dynamics. Based on this lemma, we propose two new algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), which offer fast policy transfer and dynamics modelling, respectively. RPO transfers the policy evaluated in one environment to maximize the return in another, while RTO updates the parameterized dynamics model to reduce the gap between the dynamics of the two environments. Integrating the two algorithms results in the complete Relative Policy-Transition Optimization (RPTO) algorithm, in which the policy interacts with the two environments simultaneously, such that data collections from two environments, policy and transition updates are completed in one closed loop to form a principled learning framework for policy transfer. We demonstrate the effectiveness of RPTO on a set of MuJoCo continuous control tasks by creating policy transfer problems via variant dynamics.