Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

State Regularized Policy Optimization on Data with Dynamics Shift

Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, Bo An

TL;DR

This paper tackles reinforcement learning under dynamics shift by proposing State Regularized Policy Optimization (SRPO), which leverages the insight that optimal policies often induce similar stationary state distributions across related environments. SRPO regularizes the learning process to align the agent’s stationary state distribution with a cross-dynamics optimal distribution, formulated as a KL constraint and implemented via a data-driven density-ratio surrogate learned with a GAN-style discriminator. The authors provide a lower-bound performance guarantee for homomorphous MDPs and demonstrate that SRPO improves data efficiency and overall performance when added to state-of-the-art context-based methods in both online and offline settings, supported by extensive experiments on MuJoCo and ablation studies. The results indicate SRPO’s practical potential for reusing data across diverse dynamics, reducing sample complexity, and enhancing policy robustness in real-world where dynamics vary.

Abstract

In many real-world scenarios, Reinforcement Learning (RL) algorithms are trained on data with dynamics shift, i.e., with different underlying environment dynamics. A majority of current methods address such issue by training context encoders to identify environment parameters. Data with dynamics shift are separated according to their environment parameters to train the corresponding policy. However, these methods can be sample inefficient as data are used \textit{ad hoc}, and policies trained for one dynamics cannot benefit from data collected in all other environments with different dynamics. In this paper, we find that in many environments with similar structures and different dynamics, optimal policies have similar stationary state distributions. We exploit such property and learn the stationary state distribution from data with dynamics shift for efficient data reuse. Such distribution is used to regularize the policy trained in a new environment, leading to the SRPO (\textbf{S}tate \textbf{R}egularized \textbf{P}olicy \textbf{O}ptimization) algorithm. To conduct theoretical analyses, the intuition of similar environment structures is characterized by the notion of homomorphous MDPs. We then demonstrate a lower-bound performance guarantee on policies regularized by the stationary state distribution. In practice, SRPO can be an add-on module to context-based algorithms in both online and offline RL settings. Experimental results show that SRPO can make several context-based algorithms far more data efficient and significantly improve their overall performance.

State Regularized Policy Optimization on Data with Dynamics Shift

TL;DR

This paper tackles reinforcement learning under dynamics shift by proposing State Regularized Policy Optimization (SRPO), which leverages the insight that optimal policies often induce similar stationary state distributions across related environments. SRPO regularizes the learning process to align the agent’s stationary state distribution with a cross-dynamics optimal distribution, formulated as a KL constraint and implemented via a data-driven density-ratio surrogate learned with a GAN-style discriminator. The authors provide a lower-bound performance guarantee for homomorphous MDPs and demonstrate that SRPO improves data efficiency and overall performance when added to state-of-the-art context-based methods in both online and offline settings, supported by extensive experiments on MuJoCo and ablation studies. The results indicate SRPO’s practical potential for reusing data across diverse dynamics, reducing sample complexity, and enhancing policy robustness in real-world where dynamics vary.

Abstract

In many real-world scenarios, Reinforcement Learning (RL) algorithms are trained on data with dynamics shift, i.e., with different underlying environment dynamics. A majority of current methods address such issue by training context encoders to identify environment parameters. Data with dynamics shift are separated according to their environment parameters to train the corresponding policy. However, these methods can be sample inefficient as data are used \textit{ad hoc}, and policies trained for one dynamics cannot benefit from data collected in all other environments with different dynamics. In this paper, we find that in many environments with similar structures and different dynamics, optimal policies have similar stationary state distributions. We exploit such property and learn the stationary state distribution from data with dynamics shift for efficient data reuse. Such distribution is used to regularize the policy trained in a new environment, leading to the SRPO (\textbf{S}tate \textbf{R}egularized \textbf{P}olicy \textbf{O}ptimization) algorithm. To conduct theoretical analyses, the intuition of similar environment structures is characterized by the notion of homomorphous MDPs. We then demonstrate a lower-bound performance guarantee on policies regularized by the stationary state distribution. In practice, SRPO can be an add-on module to context-based algorithms in both online and offline RL settings. Experimental results show that SRPO can make several context-based algorithms far more data efficient and significantly improve their overall performance.
Paper Structure (39 sections, 7 theorems, 39 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 39 sections, 7 theorems, 39 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Backgroud
  3. Preliminaries
  4. Related Work
  5. MDPs with Different Dynamics
  6. Behavior Regularized Methods
  7. Leveraging stationary state distributions
  8. State Regularized Policy Optimization
  9. Motivating Example
  10. State Regularized Policy Optimization
  11. Data-based Surrogate of the Density Ratio
  12. Practical Algorithm
  13. Theoretical Analysis
  14. Experiments
  15. Experiment Setup
  16. ...and 24 more sections

Key Result

Proposition 3.1

In a GAN, when the real data distribution is $\zeta(s)$ and the generated data distribution is $d_\pi(s)$, the output of the discriminator $D(s)$ follows

Figures (7)

  • Figure 1: Performance comparison of PPO schulman2017proximal, CaDM lee2020context and CaDM+SRPO in the Ant environment, where SRPO is our proposed state regularized policy optimization method. Details of the experiment setup are in Sec. \ref{['sec:exp_setup']}.
  • Figure 2: Visualization of state and action densities in data sampled from the Inverted Pendulum environment with gravity 5 and 10. Under both gravities, the state distribution has high density with low pendulum speed and small pendulum angle. Meanwhile, the action distribution has different peaks in density under different gravities.
  • Figure 3: HMM in MDP with optimality variables $\mathcal{O}_t$.
  • Figure 4: Results of online experiments on MuJoCo tasks. The comparison is made between CaDM+SRPO and baseline algorithms PPO, CaDM. Our CaDM+SRPO algorithm has the best overall performance in experiments with 3 and 5 different environment dynamics. The curves show the average return on 4 random seeds and the shadow areas reflect the standard deviation.
  • Figure 5: Comparison of values on states with high and low output of the discriminator $D$.
  • ...and 2 more figures

Theorems & Definitions (12)

  • Proposition 3.1
  • Definition 4.1: homomorphous MDPs
  • Theorem 4.2
  • Theorem 4.3
  • Lemma A.1
  • proof
  • Theorem A.2: Restatement of Thm. \ref{['cor:a_gap']}
  • proof
  • Lemma A.3
  • proof
  • ...and 2 more