The impact of intrinsic rewards on exploration in Reinforcement Learning

TL;DR

This paper tackles the hard exploration problem in sparse-reward Reinforcement Learning by evaluating how different intrinsic rewards, representing four diversity levels, affect exploration patterns. It introduces a unified taxonomy (State, State+Dynamics, Policy, Skill) and a standardized evaluation framework using PPO on MiniGrid with grid and RGB observations. The study shows State Count excels in low-dimensional state spaces, while Max Entropy offers more robust exploration in high-dimensional observations; learning diverse skills via DIAYN generally does not promote exploration in MiniGrid due to skill-space learning and discrimination biases. These insights provide practical guidance for selecting intrinsic rewards and highlight representation learning and environment dimensionality as critical factors in exploration. The work lays a foundation for broader benchmarks and more nuanced integration of representation learning with intrinsic motivation.

Abstract

One of the open challenges in Reinforcement Learning is the hard exploration problem in sparse reward environments. Various types of intrinsic rewards have been proposed to address this challenge by pushing towards diversity. This diversity might be imposed at different levels, favouring the agent to explore different states, policies or behaviours (State, Policy and Skill level diversity, respectively). However, the impact of diversity on the agent's behaviour remains unclear. In this work, we aim to fill this gap by studying the effect of different levels of diversity imposed by intrinsic rewards on the exploration patterns of RL agents. We select four intrinsic rewards (State Count, Intrinsic Curiosity Module (ICM), Maximum Entropy, and Diversity is all you need (DIAYN)), each pushing for a different diversity level. We conduct an empirical study on MiniGrid environment to compare their impact on exploration considering various metrics related to the agent's exploration, namely: episodic return, observation coverage, agent's position coverage, policy entropy, and timeframes to reach the sparse reward. The main outcome of the study is that State Count leads to the best exploration performance in the case of low-dimensional observations. However, in the case of RGB observations, the performance of State Count is highly degraded mostly due to representation learning challenges. Conversely, Maximum Entropy is less impacted, resulting in a more robust exploration, despite being not always optimal. Lastly, our empirical study revealed that learning diverse skills with DIAYN, often linked to improved robustness and generalisation, does not promote exploration in MiniGrid environments. This is because: i) learning the skill space itself can be challenging, and ii) exploration within the skill space prioritises differentiating between behaviours rather than achieving uniform state visitation.

Figures (33)

• Figure 1: Overview of the empirical study pipeline, illustrating the flow from input observations to action selection, and reward computation (both extrinsic and intrinsic) within the PPO framework.
• Figure 2: Neural Network Architectures
• Figure 3: Performance of four metrics -- Episodic Return, Observation Coverage, Agent's Position Coverage, and Entropy -- plotted against the number of transitions (frames) processed by the environment. Observations are represented as grid encodings. The results, averaged over five seeds with standard deviation shading, include evaluations across the four environments described in Section \ref{['subsection:Environment']}. The baseline model, PPO, operates without intrinsic rewards, while the other four algorithms incorporate intrinsic rewards detailed in Section \ref{['intrinsic rewards']}. For DIAYN, we differentiate between pretraining (for frames $< 25M$) and finetuning (for frames $\in [25M, 40M]$). Vertical dash-dot lines indicate the beginning of DIAYN finetuning, while horizontal dash-dot lines represent the theoretical maximum entropy of the policy, defined as $H_{max} = H(\mathcal{U}_{|A|}) = \log(|A|)$.
• Figure 4: Analogous to Figure \ref{['MatrixA']}, but observations are partial RGB images.
• Figure 5: Histogram showing the average number of frames required for each exploration method (PPO baseline and the four intrinsic rewards detailed in Section \ref{['intrinsic rewards']}) to discover rewards across the environments described in Section \ref{['subsection:Environment']}. Observations are grid encodings. Each bar is divided into three progressively fading compartments, representing the frames at which the first, second, and third rewards are collected during training, with lower values indicating better performance. Results are averaged over five runs. For variation measures alongside average results, see the tables in \ref{['sec:appendix5']}.