Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning

TL;DR

The paper tackles cross-dynamics reinforcement learning where data come from multiple evolving dynamics, arguing that imitation from expert state distributions can mislead when some states become inaccessible. It introduces ASOR, a framework that couples reward-maximization with imitation constrained to globally accessible states via an $\mathcal{F}$-distance to the expert accessible distribution, and provides two theoretical instantiations (JS divergence and network distance) plus a GAN-like practical objective. The authors prove infinite-sample and finite-sample guarantees and demonstrate substantial empirical gains across offline/online benchmarks (e.g., MuJoCo, MetaDrive, Minigrid, Fall Guys-like game), showing that ASOR enhances cross-domain policy transfer when dynamics shift. As a general add-on module, ASOR can augment existing cross-dynamics RL algorithms to improve robustness and performance, while recognizing limitations related to HiP-MDP assumptions and potential edge cases.

Abstract

To learn from data collected in diverse dynamics, Imitation from Observation (IfO) methods leverage expert state trajectories based on the premise that recovering expert state distributions in other dynamics facilitates policy learning in the current one. However, Imitation Learning inherently imposes a performance upper bound of learned policies. Additionally, as the environment dynamics change, certain expert states may become inaccessible, rendering their distributions less valuable for imitation. To address this, we propose a novel framework that integrates reward maximization with IfO, employing F-distance regularized policy optimization. This framework enforces constraints on globally accessible states--those with nonzero visitation frequency across all considered dynamics--mitigating the challenge posed by inaccessible states. By instantiating F-distance in different ways, we derive two theoretical analysis and develop a practical algorithm called Accessible State Oriented Policy Regularization (ASOR). ASOR serves as a general add-on module that can be incorporated into various RL approaches, including offline RL and off-policy RL. Extensive experiments across multiple benchmarks demonstrate ASOR's effectiveness in enhancing state-of-the-art cross-domain policy transfer algorithms, significantly improving their performance.

Figures (7)

• Figure 1: Lava world example with dynamics shift.
• Figure 2: Results of online experiments on MuJoCo and MetaDrive tasks. "NS" refers to tasks with non-stationary environment dynamics.
• Figure 3: Left: Comparisons of the logarithm of the discriminator output, i.e., the augmented reward, and the environment reward on different states in the Walker-2d environment. The augmented reward can better reflect the state optimality. Right: Curves for average extra loss and augmented reward in the fall-guys like game environment.
• Figure 4: Demonstrations of dynamics shift caused by different trampoline effect. Colors and textures are only for visual enhancement and are not part of the agent's observations.
• Figure 5: MetaDrive environments with different traffic densities.

Theorems & Definitions (16)

• Definition 3.1: Globally Accessible States
• Definition 3.2: $\mathcal{F}$-distance, Definition 2 in sanjeev2017generalization
• Definition 3.3
• Proposition 3.4
• Theorem 3.5
• Proposition 3.6: Neural network distance sanjeev2017generalization
• Theorem 3.7
• Proposition 3.8
• Lemma 1.1: Value Discrepancy
• proof