Exploration by Random Distribution Distillation

TL;DR

The paper addresses exploration in sparse-reward reinforcement learning by introducing Random Distribution Distillation (RDD), a method that samples target outputs from a state-dependent normal distribution and derives an intrinsic reward from the predictor-target discrepancy. The core contribution is a theoretically grounded framework that unifies count-based and prediction-error exploration through a dual-term bonus: a pseudo-count-like component and a predictor-discrepancy component, with rigorous analysis showing unbiased estimation and favorable variance properties. Empirically, RDD achieves faster convergence and stronger exploration across Atari, Adroit, and Fetch tasks, supported by numerical and ablation studies, and a MountainCar case study illustrating broader state visitation. The work offers a practical, scalable exploration tool with solid theoretical underpinnings and clear connections to existing methods such as RND and DRND, advancing robust online RL in high-dimensional environments.

Abstract

Exploration remains a critical challenge in online reinforcement learning, as an agent must effectively explore unknown environments to achieve high returns. Currently, the main exploration algorithms are primarily count-based methods and curiosity-based methods, with prediction-error methods being a prominent example. In this paper, we propose a novel method called \textbf{R}andom \textbf{D}istribution \textbf{D}istillation (RDD), which samples the output of a target network from a normal distribution. RDD facilitates a more extensive exploration by explicitly treating the difference between the prediction network and the target network as an intrinsic reward. Furthermore, by introducing randomness into the output of the target network for a given state and modeling it as a sample from a normal distribution, intrinsic rewards are bounded by two key components: a pseudo-count term ensuring proper exploration decay and a discrepancy term accounting for predictor convergence. We demonstrate that RDD effectively unifies both count-based and prediction-error approaches. It retains the advantages of prediction-error methods in high-dimensional spaces, while also implementing an intrinsic reward decay mode akin to the pseudo-count method. In the experimental section, RDD is compared with more advanced methods in a series of environments. Both theoretical analysis and experimental results confirm the effectiveness of our approach in improving online exploration for reinforcement learning tasks.

Figures (6)

• Figure 1: Numerical Example. We investigated the decay of bonus with each access to the same state when the bonus remains constant, as well as the bonus obtained by the agent through random wandering (simulating a random strategy) on the mini-dataset.
• Figure 2: Main experiment. We selected the currently most effective prediction-error method on 12 different environments for the experiment. Most of the environments we used were extremely sparse reward.
• Figure 3: Visualize the exploration process on the MountainCar. In the MountainCar environment, the position of the car on the x-axis can reflect the success of the task. We represent the degree of exploration through the probability density of the agents of each method on the x-axis at different steps.
• Figure 4: We mainly analyzed the mean and variance of the target network, as well as the influence of the output dimensions in different environments on the exploration effect.
• Figure 5: Environments of experiments

Theorems & Definitions (3)

• Lemma 4.1
• Corollary 4.2
• Theorem 4.3