Maximum Entropy Hindsight Experience Replay

TL;DR

This work shows that the previous PPO-HER algorithm can be improved by selectively applying Hindsight experience replay in a principled manner.

Abstract

Hindsight experience replay (HER) is well-known to accelerate goal-based reinforcement learning (RL). While HER is generally applied to off-policy RL algorithms, we previously showed that HER can also accelerate on-policy algorithms, such as proximal policy optimization (PPO), for goal-based Predator-Prey environments. Here, we show that we can improve the previous PPO-HER algorithm by selectively applying HER in a principled manner.

Figures (11)

• Figure 1: Predator-prey environments.
• Figure 2: Learning curves for agents trained using S-ratios in the range $[0, 1]$. Across conditions, S-ratios in the range $[0.5, 0.7]$ tend to perform best, with a slight bias towards higher S-ratios. Bold lines and shaded regions represent medians and interquartile ranges, respectively.
• Figure 3: $M_c$ metrics for S-ratios in the range $[0, 1]$. The shape of the bar plots, rather than the magnitudes of the columns, should be interpreted. Information theory predicts a parabolic shape to the bar plot, maximized at an S-ratio of 0.5, with $M_c$ values of 0 for S-ratios of 0 and 1. The actual shapes of the bar plots differ somewhat from parabolas, and higher S-ratios tend to perform better than predicted.
• Figure 4: Learning curves for agents trained using targeted MEHER, where targets for artificial failures are generated near the target, rather than far away. Comparing to un-targeted MEHER (Figure \ref{['fig:meher']}), targeted MEHER resulted in very few changes.
• Figure 5: $M_c$ metrics for S-ratios in the range $[0, 1]$ when using targeted MEHER. The shape of the bar plots, rather than the magnitudes of the columns, should be interpreted. Information theory predicts a parabolic shape to the bar plot, maximized at a S-ratio of 0.5, with $M_c$ values of 0 for S-ratios of 0 and 1. The actual shapes of the bar plots differ somewhat, and higher S-ratios tend to perform better than predicted. The shapes of the bar plots look similar to the bar plots for un-targeted MEHER (Figure \ref{['fig:meherDist']}).