Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Divergence-Augmented Policy Optimization

Qing Wang, Yingru Li, Jiechao Xiong, Tong Zhang

TL;DR

Empirical experiments show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, the method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.

Abstract

In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.

Divergence-Augmented Policy Optimization

TL;DR

Empirical experiments show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, the method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.

Abstract

In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.
Paper Structure (24 sections, 2 theorems, 46 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 24 sections, 2 theorems, 46 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Process
  4. Bregman Divergence
  5. Method
  6. Policy Optimization and Mirror Descent
  7. Parametric Policy-based Algorithm
  8. Off-policy Correction
  9. Related Works
  10. Experiments
  11. Algorithm Settings
  12. Empirical Results
  13. Divergence augmentation vs Entropy augmentation
  14. Multi-step divergence vs 1-step divergence
  15. Conclusion
  16. ...and 9 more sections

Key Result

Lemma 1

(Policy Gradient Theorem sutton2000policy) For $d_\pi$ and $\mu_\pi$ defined previously, the following equation holds for any state-action function $f: \mathcal{S}\times\mathcal{A} \rightarrow \mathbb{R}$: where $\mathcal{Q}^\pi$ is defined as an operator such that

Figures (7)

  • Figure 1: Relative score improvement of PPO+DA compared with PPO on 58 Atari environments. The relative performance is calculated as a $\frac{\text{proposed}-\text{baseline}}{\max(\text{human},\text{baseline})-\text{random}}$wang2016dueling. The Atari games are categorized according to Figure 4 of oh2018self.
  • Figure 2: Performance comparison of selected environments of Atari games. The performance of PPO, PPO+DA, PPO+DA (1-step), and PPO+Entropy are plotted in different colors. The score for each game is plotted on the y-axis with running time on the x-axis, as the algorithm is paralleled asynchronously in a distributed environment. For each line in the plots, we run the experiment 5 times with the same parameters and environment settings. The median scores are plotted in solid lines, while the regions between 25% and 75% quantiles are shaded with respective colors.
  • Figure 3: Performance comparison of PPO+DA with PPO on 58 Atari games. Each experiment is allowed to run for 2 hours as a limited time.
  • Figure 4: Performance comparison of PPO+DA with PPO on 58 Atari games, with the number of used actors increased to 64 and running time increased to 4 hours.
  • Figure :
  • ...and 2 more figures

Theorems & Definitions (3)

  • Lemma 1
  • Proposition 1
  • proof