Adventurer: Exploration with BiGAN for Deep Reinforcement Learning

TL;DR

Adventurer tackles sparse rewards and high-dimensional observation challenges in deep reinforcement learning by introducing a BiGAN-based state novelty estimator that produces an intrinsic reward to drive exploration. The novelty is captured by a weighted combination of pixel-level reconstruction error and discriminator feature-matching error, embedded into a normalized intrinsic reward and integrated with PPO via augmented advantages. Empirical results on continuous robotic tasks and Atari-style image tasks show Adventurer achieves competitive or superior exploration performance compared to baselines like PPO and RND, with improved gains under resettable environments using episodic memory. This approach provides a flexible, high-dimensional novelty signal that can augment various policy optimization methods and enhance sample efficiency in complex environments.

Abstract

Recent developments in deep reinforcement learning have been very successful in learning complex, previously intractable problems. Sample efficiency and local optimality, however, remain significant challenges. To address these challenges, novelty-driven exploration strategies have emerged and shown promising potential. Unfortunately, no single algorithm outperforms all others in all tasks and most of them struggle with tasks with high-dimensional and complex observations. In this work, we propose Adventurer, a novelty-driven exploration algorithm that is based on Bidirectional Generative Adversarial Networks (BiGAN), where BiGAN is trained to estimate state novelty. Intuitively, a generator that has been trained on the distribution of visited states should only be able to generate a state coming from the distribution of visited states. As a result, novel states using the generator to reconstruct input states from certain latent representations would lead to larger reconstruction errors. We show that BiGAN performs well in estimating state novelty for complex observations. This novelty estimation method can be combined with intrinsic-reward-based exploration. Our empirical results show that Adventurer produces competitive results on a range of popular benchmark tasks, including continuous robotic manipulation tasks (e.g. Mujoco robotics) and high-dimensional image-based tasks (e.g. Atari games).

Figures (5)

• Figure 1: Adventurer architecture. Blue: the structure of BiGAN; Green: the workflow of state novelty estimation. The BiGAN is trained on visited states. The novelty of an input state $s$ is estimated by $\alpha L_{G}(s)+(1-\alpha)L_{D}(s)$.
• Figure 2: Novelty score on CIFAR-10. The dashed line represents the novelty score that the BiGAN is trained only on image 0 and tested with image 0.
• Figure 3: Setting 1: the novelty score distributions for novelty estimators $B(S)$, $L_G(s)$, $L_D(S)$, RND,and VAE, which are trained with observations $D_{1a}$.
• Figure 4: The mean success rate for both Fetch arm and Shadow Dexterous hand tasks. The x-axis is the number of training samples.
• Figure 5: The game score for Montezuma’s Revenge, Gravitar, and Solaris. The x-axis is the number of training frames.