Episodic Curiosity through Reachability

TL;DR

Sparse rewards limit reinforcement learning; this work introduces episodic curiosity (EC) based on reachability to past observations stored in episodic memory. A reachability network (R-network) predicts how many steps separate observations, guiding a novelty bonus b that is added to task rewards during PPO training, with memory kept under a fixed capacity and novelty threshold. Across ViZDoom, DMLab, and MuJoCo, EC outperforms the prior state-of-the-art ICM on sparse tasks, avoids couch-potato behavior, and enables first-person-view curiosity to drive locomotion in MuJoCo. The approach is robust to stochastic environments and supported by extensive ablations, generalization studies, and computational considerations, highlighting memory-guided exploration as a powerful tool for sparse-reward RL.

Abstract

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

Figures (9)

• Figure 1: We define novelty through reachability. The nodes in the graph are observations, the edges --- possible transitions. The blue nodes are already in memory, the green nodes are reachable from the memory within $k = 2$ steps (not novel), the orange nodes are further away --- take more than $k$ steps to reach (novel). In practice, the full possible transition graph is not available, so we train a neural network approximator to predict if the distance in steps between observations is larger or smaller than $k$.
• Figure 2: Left: siamese architecture of reachability (R) network. Right: R-network is trained based on a sequence of observations that the agent encounters while acting. The temporally close (within threshold) pairs of observations are positive examples, while temporally far ones --- negatives.
• Figure 3: The use of episodic curiosity (EC) module for reward bonus computation. The module take a current observation as input and computes a reward bonus which is higher for novel observations. This bonus is later summed up with the task reward and used for training an RL agent.
• Figure 4: Examples of tasks considered in our experiments: (a) VizDoom static maze goal reaching, (b) DMLab randomized maze goal reaching, (c) DMLab key-door puzzle, (d) MuJoCo ant locomotion out of first-person-view curiosity.
• Figure 5: Examples of maze types used in our experiments: (a) VizDoom static maze goal reaching, (b) DMLab randomized maze goal reaching, (c) DMLab randomized maze goal reaching with doors.