Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory

Chenyang Miao

TL;DR

FEMA explicitly stores short-horizon failure experiences through an episodic memory module and retrieves similar failure experiences during interactions and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value.

Abstract

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.

Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory

TL;DR

FEMA explicitly stores short-horizon failure experiences through an episodic memory module and retrieves similar failure experiences during interactions and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value.

Abstract

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.
Paper Structure (14 sections, 7 equations, 12 figures, 1 table)

This paper contains 14 sections, 7 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. DRL and Experience Reuse
  4. Episodic Control
  5. Method
  6. Overview
  7. Failure Episodic Memory Construction
  8. Risk-aware Action Selection Mechanism
  9. Experiments
  10. Experiment Settings
  11. Simulation Experiments
  12. Real-world Experiments
  13. Ablation Study
  14. Conclusion

Figures (12)

  • Figure 1: Robotic agents frequently experience premature terminations during training. FEMA retrieves past similar failure experiences from the episodic memory to guide the robot away from previously experienced hazardous states, promoting longer-horizon exploration.
  • Figure 2: Framework of Failure Episodic Memory Alert (FEMA)
  • Figure 3: Joint state-action encoder $\varphi(s,a)$
  • Figure 4: MuJoCo evaluation tasks:(a)Humanoid (b)Walker2d (c)Hopper (d)Ant
  • Figure 5: Learning curves of EMAC, SAC, SAC+FEMA on MuJoCo benchmark tasks.
  • ...and 7 more figures