MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning

TL;DR

MOBODY, a Model-Based Off-Dynamics Offline RL algorithm that optimizes a policy using learned target dynamics transitions to explore the target domain, rather than only being trained with the low dynamics-shift transitions, is proposed.

Abstract

We study off-dynamics offline reinforcement learning, where the goal is to learn a policy from offline source and limited target datasets with mismatched dynamics. Existing methods either penalize the reward or discard source transitions occurring in parts of the transition space with high dynamics shift. As a result, they optimize the policy using data from low-shift regions, limiting exploration of high-reward states in the target domain that do not fall within these regions. Consequently, such methods often fail when the dynamics shift is significant or the optimal trajectories lie outside the low-shift regions. To overcome this limitation, we propose MOBODY, a Model-Based Off-Dynamics Offline RL algorithm that optimizes a policy using learned target dynamics transitions to explore the target domain, rather than only being trained with the low dynamics-shift transitions. For the dynamics learning, built on the observation that achieving the same next state requires taking different actions in different domains, MOBODY employs separate action encoders for each domain to encode different actions to the shared latent space while sharing a unified representation of states and a common transition function. We further introduce a target Q-weighted behavior cloning loss in policy optimization to avoid out-of-distribution actions, which push the policy toward actions with high target-domain Q-values, rather than high source domain Q-values or uniformly imitating all actions in the offline dataset. We evaluate MOBODY on a wide range of MuJoCo and Adroit benchmarks, demonstrating that it outperforms state-of-the-art off-dynamics RL baselines as well as policy learning methods based on different dynamics learning baselines, with especially pronounced improvements in challenging scenarios where existing methods struggle.

Figures (4)

• Figure 1: Comparison between DARA liu2022dara (a SOTA model-free reward regularization method for offline off-dynamics RL), MOPO yu2020mopo (a vanilla model-based offline RL), and MOBODY on two MuJoCo tasks. We show that 1) the model-free method DARA receives low reward compared with model-based MOBODY due to a lack of exploration in the target domain, and 2) MOPO fails as it cannot learn a good transition for exploration with a combined source and target dataset.
• Figure 2: Architecture of the dynamics model. MOBODY encodes the state with $\phi_E$ and state action with $\psi$, outputs the next state through $\phi_T$, and learns the dynamics for both domains by transition loss shown in purple double arrow$\textcolor{myviolet!90!black}{\Leftrightarrow}$. It learns the state action representation by matching the state action representation $z_{sa}$ with the next state representation $z_{s'}$ through encoder loss shown in the green double arrow$\textcolor{green!50!black}{\Leftrightarrow}$ and the state representation through cycle transition loss shown in orange double arrow$\textcolor{orange!70!black}{\Leftrightarrow}$.
• Figure 3: Aggregation experimental results on MuJoco kinematic and morphology shift task, and Manipulation tasks. Our method outperforms the baselines. Detailed results of each environment, shift type, and shift level are referred to \ref{['exp: additional mujoco']} and \ref{['exp: additional manipulation']} in the \ref{['sec: additional exp']}.
• Figure 4: Performance of our MOBODY and baselines in different dynamics shift with various shift levels $\{0.1, 0.5, 2.0, 5.0 \}$. The scores are summed over all the environments (HalfCheetah, Ant, Walker2D, and Hopper) in the target domain. We directly compare the algorithms in the same dynamics shift levels. The higher scores indicate better performance. We can observe a larger improvement for larger shift cases (0.1 and 5.0).