Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Diyuan Shi, Yiqi Tang, Zifeng Zhuang, Donglin Wang

Abstract

Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Abstract

Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.
Paper Structure (22 sections, 7 equations, 11 figures, 4 tables, 1 algorithm)

This paper contains 22 sections, 7 equations, 11 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background
  4. Method
  5. Experiments
  6. Q1: How do various factors influence NFPO's performance?
  7. Q2: How is NFPO's performance compared to SOTA PPO implementations?
  8. Q3: Does NFPO learn multi-modal behaviors?
  9. Q4: How is NFPO compared to other multi-modal policy learning methods?
  10. Q5: Could NFPO's policy transfer to real-world robotic platforms?
  11. Conclusion
  12. Appendix
  13. Determination of values in \ref{['tbl:algo_compare']}
  14. Analysis on the stability of tanh vs clip
  15. Additional experiments on Mujoco
  16. ...and 7 more sections

Figures (11)

  • Figure 2: Training NFs in Unitree Gym's g1. The experiment is in 3 seeds with 95% confidence interval. s_none early stopped due to training instability, s_clip and s_tanh overlaps in determinant plot.
  • Figure 3: Studies in various factors. Experiments are in 3 seeds with 95% confidence interval.
  • Figure 4: Studies in various factors. Experiments are in 3 seeds with 95% confidence interval.
  • Figure 5: Learning curves of various methods on representative robotic tasks. The experiments are in 10 seeds. The errorbars are 95% confidence interval.
  • Figure 6: Difference in generated trajectory of NFPO and PPO in gridworld.
  • ...and 6 more figures