Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradients

Luca Della Libera

Abstract

Recent advances in deep reinforcement learning have achieved impressive results in a wide range of complex tasks, but poor sample efficiency remains a major obstacle to real-world deployment. Soft actor-critic (SAC) mitigates this problem by combining stochastic policy optimization and off-policy learning, but its applicability is restricted to distributions whose gradients can be computed through the reparameterization trick. This limitation excludes several important examples such as the beta distribution, which was shown to improve the convergence rate of actor-critic algorithms in high-dimensional continuous control problems thanks to its bounded support. To address this issue, we investigate the use of implicit reparameterization, a powerful technique that extends the class of reparameterizable distributions. In particular, we use implicit reparameterization gradients to train SAC with the beta policy on simulated robot locomotion environments and compare its performance with common baselines. Experimental results show that the beta policy is a viable alternative, as it outperforms the normal policy and is on par with the squashed normal policy, which is the go-to choice for SAC. The code is available at https://github.com/lucadellalib/sac-beta.

Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradients

Abstract

Recent advances in deep reinforcement learning have achieved impressive results in a wide range of complex tasks, but poor sample efficiency remains a major obstacle to real-world deployment. Soft actor-critic (SAC) mitigates this problem by combining stochastic policy optimization and off-policy learning, but its applicability is restricted to distributions whose gradients can be computed through the reparameterization trick. This limitation excludes several important examples such as the beta distribution, which was shown to improve the convergence rate of actor-critic algorithms in high-dimensional continuous control problems thanks to its bounded support. To address this issue, we investigate the use of implicit reparameterization, a powerful technique that extends the class of reparameterizable distributions. In particular, we use implicit reparameterization gradients to train SAC with the beta policy on simulated robot locomotion environments and compare its performance with common baselines. Experimental results show that the beta policy is a viable alternative, as it outperforms the normal policy and is on par with the squashed normal policy, which is the go-to choice for SAC. The code is available at https://github.com/lucadellalib/sac-beta.
Paper Structure (12 sections, 14 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 12 sections, 14 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Process
  4. Soft Actor-Critic
  5. Implicit Reparameterization Gradients
  6. Method
  7. Experiments
  8. MuJoCo Environments
  9. Training Setup and Hyperparameters
  10. Implementation and Hardware
  11. Results and Discussion
  12. Conclusion

Figures (2)

  • Figure 1: Performance comparison on the four considered MuJoCo environments. Curves represent the average return $\pm$$2$ standard deviations over $5$ random seeds, smoothed using a centered moving average with radius $1$.
  • Figure 2: Ablation study for SAC-Beta-OMT on Ant-v4. Curves represent the average return $\pm$$2$ standard deviations over $5$ random seeds, smoothed using a centered moving average with radius $1$.