First return, then explore

TL;DR

Go-Explore tackles sparsity and deception in reinforcement learning by separating memory (archive of visited states) from exploration, using a first-return then explore paradigm to avoid detachment and derailment. It shows that explicit state remembering and returning to high-potential states enables complete, robust exploration, solving all hard-exploration Atari games and surpassing prior methods on Montezuma’s Revenge and Pitfall, with a robotics demonstration on sparse rewards. The work also extends to policy-based Go-Explore, enabling exploration under stochastic environments and improving sample efficiency, while analyzing differences with DTSIL and Agent57. Its results suggest that simple, general principles of remembering states and returning before exploring can drive substantial gains in diverse domains.

Abstract

The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states ("detachment") and from failing to first return to a state before exploring from it ("derailment"). We introduce Go-Explore, a family of algorithms that addresses these two challenges directly through the simple principles of explicitly remembering promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games, with orders of magnitude improvements on the grand challenges Montezuma's Revenge and Pitfall. We also demonstrate the practical potential of Go-Explore on a sparse-reward pick-and-place robotics task. Additionally, we show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handle stochasticity throughout training. The substantial performance gains from Go-Explore suggest that the simple principles of remembering states, returning to them, and exploring from them are a powerful and general approach to exploration, an insight that may prove critical to the creation of truly intelligent learning agents.

Figures (14)

• Figure 1: Overview of Go-Explore.(a) Probabilistically select a state from the archive, guided by heuristics that prefer states associated with promising cells. (b) Return to the selected state, such as by restoring simulator state or by running a goal-conditioned policy. (c) Explore from that state by taking random actions or sampling from a policy. (d) Map every state encountered during returning and exploring to a low-dimensional cell representation. (e) Add states that map to new cells to the archive and update other archive entries.
• Figure 1: Neural network architectures.(a) The Atari architecture is based on the architecture provided with the backward algorithm implementation. The input consists of the RGB channels of the last four frames (re-scaled to 80 by 105 pixels) concatenated, resulting in 12 input channels. The network consists of 3 convolutional layers, 2 fully connected layers, and a layer of Gated Recurrent Units (GRUs)cho2014properties. The network has a policy head $\pi(s|a)$ and a value head $V(s)$. (b) For the robotics problem, the architecture consists of two separate networks, each with 2 fully connected layers and a GRU layer. One network specifies the policy $\pi(s|a)$ by returning a mean $\mu$ and variance $\sigma$ for the actuator torques of the arm and the desired position of each of the two fingers of the gripper (gripper fingers are implemented as Mujoco position actuatorstodorov2012mujoco with $kp=10^4$ and a control range of $[0, 0.05]$). The other network implements the value function $V(s)$. (c) The architecture for policy-based Go-Explore is identical to the Atari architecture, except that the goal representation $g$ is concatenated with the input of the first fully connected layer.
• Figure 1: Example of detachment with intrinsic reward. Green areas indicate intrinsic reward, white indicates areas where no intrinsic reward remains, and purple areas indicate where the algorithm is currently exploring.
• Figure 2: Performance of robustified Go-Explore on Atari games. (a) Go-Explore produces massive improvements over previous methods on Montezuma's Revenge, a grand challenge which was the focus of intense research for many years. (b) It exceeds the average human score in each of the 11 hard-exploration and unsolved games in the Atari suite, and matches or beats (often by a factor of 2 or more) the state of the art in each of these games.
• Figure 2: Maximum end-of-episode score found by the exploration phase on Atari. Because only scores achieved at the episode end are reported, the plots for some games (e.g. Solaris) begin after the start of the run, when the episode end is first reached. In (a), averaging is over 50 runs for the 11 focus games and 5 runs for other games. In (b), averaging is over 100 runs.