Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay

Gabriel Romio, Mateus Begnini Melchiades, Bruno Castro da Silva, Gabriel de Oliveira Ramos

TL;DR

Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals that generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task.

Abstract

Hierarchical Reinforcement Learning (HRL) frameworks like Option-Critic (OC) and Multi-updates Option Critic (MOC) have introduced significant advancements in learning reusable options. However, these methods underperform in multi-goal environments with sparse rewards, where actions must be linked to temporally distant outcomes. To address this limitation, we first propose MOC-HER, which integrates the Hindsight Experience Replay (HER) mechanism into the MOC framework. By relabeling goals from achieved outcomes, MOC-HER can solve sparse reward environments that are intractable for the original MOC. However, this approach is insufficient for object manipulation tasks, where the reward depends on the object reaching the goal rather than on the agent's direct interaction. This makes it extremely difficult for HRL agents to discover how to interact with these objects. To overcome this issue, we introduce Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals. In addition to relabeling goals based on the object's final state (standard HER), 2HER also generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task. Experimental results in robotic manipulation environments show that MOC-2HER achieves success rates of up to 90%, compared to less than 11% for both MOC and MOC-HER. These results highlight the effectiveness of our dual objective relabeling strategy in sparse reward, multi-goal tasks.

Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay

TL;DR

Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals that generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task.

Abstract

Hierarchical Reinforcement Learning (HRL) frameworks like Option-Critic (OC) and Multi-updates Option Critic (MOC) have introduced significant advancements in learning reusable options. However, these methods underperform in multi-goal environments with sparse rewards, where actions must be linked to temporally distant outcomes. To address this limitation, we first propose MOC-HER, which integrates the Hindsight Experience Replay (HER) mechanism into the MOC framework. By relabeling goals from achieved outcomes, MOC-HER can solve sparse reward environments that are intractable for the original MOC. However, this approach is insufficient for object manipulation tasks, where the reward depends on the object reaching the goal rather than on the agent's direct interaction. This makes it extremely difficult for HRL agents to discover how to interact with these objects. To overcome this issue, we introduce Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals. In addition to relabeling goals based on the object's final state (standard HER), 2HER also generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task. Experimental results in robotic manipulation environments show that MOC-2HER achieves success rates of up to 90%, compared to less than 11% for both MOC and MOC-HER. These results highlight the effectiveness of our dual objective relabeling strategy in sparse reward, multi-goal tasks.
Paper Structure (12 sections, 2 equations, 10 figures, 2 tables, 1 algorithm)

This paper contains 12 sections, 2 equations, 10 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related work
  4. Method
  5. Algorithm
  6. Implementing HER into MOC
  7. Experiments
  8. Methodology
  9. Numerical results
  10. Conclusion
  11. Implementation details
  12. Additional quantitative results

Figures (10)

  • Figure 1: Comparison of different approaches with 2, 4 and 8 options in FetchReach over 100 iterations. Solid line and shaded region show the mean and standard deviation of success rates across 10 seeds.
  • Figure 2: Utilization rates for each option in the 4-option configuration for the FetchReach task, over 100 iterations. Solid line and shaded region show the mean and standard deviation across 10 seeds.
  • Figure 3: Comparison of different approaches with 2, 4 and 8 options in FetchPush over $1.5\times10^{3}$ iterations. Solid line and shaded region show the mean and standard deviation of success rates across 5 seeds. Results are averaged over a window size of 20 iterations.
  • Figure 4: Comparison of different approaches with 2, 4 and 8 options in FetchSlide over $1.5\times10^{3}$ iterations. Solid line and shaded region show the mean and standard deviation of success rates across 5 seeds. Results are averaged over a window size of 20 iterations.
  • Figure 5: Comparison of different approaches with 2, 4 and 8 options in FetchPickAndPlace over $1.5\times10^{3}$ iterations. Solid line and shaded region show the mean and standard deviation of success rates across 5 seeds. Results are averaged over a window size of 20 iterations.
  • ...and 5 more figures