Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning

TL;DR

The paper tackles the practical bottleneck of resetting real-world RL tasks by introducing Leave No Trace, a framework that jointly learns forward and reset policies. It uses uncertainty-aware value estimates to trigger early aborts, thereby avoiding irreversible states and reducing manual resets. The approach yields automatic curriculum generation and safer exploration across several continuous control tasks, with ensembles improving robustness. This work advances real-world RL by enabling longer, safer autonomous learning without frequent human intervention.

Abstract

Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.

Figures (15)

• Figure 1: Early aborts in gridworld.
• Figure 3: Early abort threshold: In our didactic example, increasing the early abort threshold causes more cautious exploration (left) without severely increasing the number of steps to solve (right).
• Figure 4: We compare our method to a non-episodic ("forward-only") approach on ball in cup. Although neither uses hard resets, only our method learns to catch the ball. As an upper bound, we also show the "status quo" approach that performs a hard reset after episode, which is often impractical outside simulation.
• Figure 5: Our method achieves equal or better rewards than the status quo with fewer manual resets.
• Figure 6: Early abort threshold: Increasing the early abort threshold to act more cautiously avoids many hard resets, indicating that early aborts help avoid irreversible states.