Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Delayed Policy Optimization

Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, Chao Huang

TL;DR

This work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem, and provides a theoretical analysis of VDPO in terms of sample complexity and performance.

Abstract

In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). However, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks often suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve learning efficiency without sacrificing performance, this work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\% less amount of samples) in the MuJoCo benchmark.

Variational Delayed Policy Optimization

TL;DR

This work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem, and provides a theoretical analysis of VDPO in terms of sample complexity and performance.

Abstract

In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). However, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks often suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve learning efficiency without sacrificing performance, this work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\% less amount of samples) in the MuJoCo benchmark.
Paper Structure (29 sections, 13 theorems, 30 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 29 sections, 13 theorems, 30 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. MDP.
  4. Delayed MDP.
  5. Variational RL.
  6. Our Approach: Variational Delayed Policy Optimization
  7. Delayed RL as Variational Inference
  8. Maximizing the performance of reference policy by TD Learning
  9. Minimizing the behaviour difference by Behaviour Cloning
  10. Theoretical Property Analysis
  11. Sample Complexity Analysis.
  12. Performance Analysis.
  13. VDPO Implementation
  14. Experimental Results
  15. Experiment Settings
  16. ...and 14 more sections

Key Result

Lemma 3.1

Let $\mathcal{M}_1, \mathcal{M}_2$ be two constant delayed MDPs with respective delays $\Delta_1, \Delta_2 (\Delta_1 < \Delta_2)$. For the optimal policies in $\mathcal{M}_1, \mathcal{M}_2$, we have $\mathcal{J}^*_1 \geq \mathcal{J}^*_2$.

Figures (4)

  • Figure 1: The training pipeline of VDPO.
  • Figure 2: Learning curves in MuJoCo tasks with 5 constant delays where the shaded areas represented the standard deviation.
  • Figure 3: Learning curves in MuJoCo tasks with 25 constant delays where the shaded areas represented the standard deviation.
  • Figure 4: Learning curves in MuJoCo tasks with 50 constant delays where the shaded areas represented the standard deviation.

Theorems & Definitions (18)

  • Lemma 3.1: Performance in delayed MDP, Theorem 4.3.1 in liotet2023delays
  • Lemma 3.2: Sample complexity of model-based policy iteration, Theorem 2 in gheshlaghi2013minimax
  • Proposition 3.3: State-level KL divergence, proof in \ref{['appendix::traj_state_kl']}
  • Lemma 3.4: Sample complexity of behaviour cloning, Theorem 15.3 in agarwal2019reinforcement
  • Lemma 3.5: Sample complexity of VDPO, proof in \ref{['appendix::sc_analysis']}
  • Proposition 3.6: Sample complexity comparison, proof in \ref{['appendix::sc_compare']}
  • Lemma 3.7: Convergence of delayed policy in VDPO, proof in \ref{['appendix::convege_p']}
  • Proposition 3.8: Consistent fixed point, proof in \ref{['appendix::converge_consistent']}
  • Proposition C.1: State-level KL divergence
  • proof
  • ...and 8 more