Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm

Wensong Bai, Chao Zhang, Yichao Fu, Peilin Zhao, Hui Qian, Bin Dai

TL;DR

PACER is the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager and demonstrates the superiority of the algorithm over the state-of-the-art.

Abstract

In this paper, we propose the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager. Specifically, the push-forward operator is leveraged in both the critic and actor to model the return distributions and stochastic policies respectively, enabling them with equal modeling capability and thus enhancing the synergetic performance. Since it is infeasible to obtain the density function of the push-forward policies, novel sample-based regularizers are integrated in the encourager to incentivize efficient exploration and alleviate the risk of trapping into local optima. Moreover, a sample-based stochastic utility value policy gradient is established for the push-forward policy update, which circumvents the explicit demand of the policy density function in existing REINFORCE-based stochastic policy gradient. As a result, PACER fully utilizes the modeling capability of the push-forward operator and is able to explore a broader class of the policy space, compared with limited policy classes used in existing distributional actor critic algorithms (i.e. Gaussians). We validate the critical role of each component in our algorithm with extensive empirical studies. Experimental results demonstrate the superiority of our algorithm over the state-of-the-art.

PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm

TL;DR

PACER is the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager and demonstrates the superiority of the algorithm over the state-of-the-art.

Abstract

In this paper, we propose the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager. Specifically, the push-forward operator is leveraged in both the critic and actor to model the return distributions and stochastic policies respectively, enabling them with equal modeling capability and thus enhancing the synergetic performance. Since it is infeasible to obtain the density function of the push-forward policies, novel sample-based regularizers are integrated in the encourager to incentivize efficient exploration and alleviate the risk of trapping into local optima. Moreover, a sample-based stochastic utility value policy gradient is established for the push-forward policy update, which circumvents the explicit demand of the policy density function in existing REINFORCE-based stochastic policy gradient. As a result, PACER fully utilizes the modeling capability of the push-forward operator and is able to explore a broader class of the policy space, compared with limited policy classes used in existing distributional actor critic algorithms (i.e. Gaussians). We validate the critical role of each component in our algorithm with extensive empirical studies. Experimental results demonstrate the superiority of our algorithm over the state-of-the-art.
Paper Structure (30 sections, 4 theorems, 53 equations, 9 figures, 8 tables, 3 algorithms)

This paper contains 30 sections, 4 theorems, 53 equations, 9 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Distributional reinforcement learning
  4. Efficient exploration policy
  5. Utility functions in DRL.
  6. Preliminaries
  7. Distributional Bellman equation
  8. The IQN and distributional TD Learning
  9. Methodology
  10. The Actor-Critic-Encourager framework
  11. Actor with push-forward policy
  12. Critic with quantile return representation
  13. Encourager with sample-based metric
  14. Stochastic utility value policy gradient
  15. Discussion
  16. ...and 15 more sections

Key Result

Theorem 1

Let $Z_{\psi}^{\pi}$ be the random return with utility of policy $\pi$, and $Q_{\psi}^{\pi}(s,a)$ be its expectation. They satisfy the following equations:

Figures (9)

  • Figure 1: The framework of PACER, where contents with identical color belong to a same module. The actor makes decisions according to a push-forward policy. The critic models return distributions with an IQN and evaluates the policy via a utility function. The encourager stimulates exploration by a sample-based regularizer.
  • Figure 2: Six MuJoCo continuous control environments. (a) Ant, (b) Walker2d, (c) HumanoidStandup, (d) Humanoid, (e) HalfCheetah, (f) Hopper.
  • Figure 3: Visualization of the navigation task.
  • Figure 4: Learning curves for PACER and baselines with $\pm$ 1 std shaded on MuJoCo continuous control tasks.
  • Figure 5: Learning curves for ablation studies.
  • ...and 4 more figures

Theorems & Definitions (10)

  • Definition 1: Push-forward operator peyre2019computational
  • Remark 1: Implicit quantile distribution
  • Theorem 1
  • Theorem 2: Stochastic utility value policy gradient
  • proof
  • Theorem 3: Stochastic Utility Value Policy Gradient (SUVPG)
  • proof
  • Corollary 1: Stochastic Value Policy Gradient
  • proof
  • Definition 2