Deep Reinforcement Learning with Hybrid Intrinsic Reward Model

TL;DR

This work tackles sparse and delayed extrinsic rewards in reinforcement learning by proposing Hybrid Intrinsic Reward (HIRE), a modular framework that blends multiple intrinsic rewards through four fusion strategies (Summation, Product, Cycle, Maximum). By enabling arbitrary numbers and types of intrinsic signals, HIRE systematically analyzes how fusion structure and reward quantity affect exploration and skill acquisition, demonstrating robust performance improvements on MiniGrid, Procgen, and URL benchmarks. Key findings show Cycle as the most robust fusion strategy, NGU+RE3 (often with ICM) as a strong combination, and that a moderate number of rewards balances exploration benefits with computational cost. The work provides practical guidance for designing intrinsically motivated agents and highlights potential for improved pre-training in unsupervised RL, with broader implications for robust exploration in dynamic environments.

Abstract

Intrinsic reward shaping has emerged as a prevalent approach to solving hard-exploration and sparse-rewards environments in reinforcement learning (RL). While single intrinsic rewards, such as curiosity-driven or novelty-based methods, have shown effectiveness, they often limit the diversity and efficiency of exploration. Moreover, the potential and principle of combining multiple intrinsic rewards remains insufficiently explored. To address this gap, we introduce HIRE (Hybrid Intrinsic REward), a flexible and elegant framework for creating hybrid intrinsic rewards through deliberate fusion strategies. With HIRE, we conduct a systematic analysis of the application of hybrid intrinsic rewards in both general and unsupervised RL across multiple benchmarks. Extensive experiments demonstrate that HIRE can significantly enhance exploration efficiency and diversity, as well as skill acquisition in complex and dynamic settings.

Figures (14)

• Figure 1: The overview of the HIRE framework. (a) Four reward fusion strategies implemented in HIRE. (b) HIRE is designed to be fully modular and decoupled from the RL training loop and can be integrated seamlessly with arbitrary RL algorithms.
• Figure 2: Screenshots of the experiment environments. (a) From left to right: KeyCorridorS10R7, MultiRoom-N12-S10, LockedRoom, and Dynamic-Obstacles-16×16. (b) Eight navigation and exploration environments from the Procgen benchmark. (c) ALE-5.
• Figure 3: Strategy-level performance comparison on the MiniGrid and Procgen benchmarks. Here, each strategy corresponds to eleven reward candidates listed in Table \ref{['tb:sum candidates']}. Bars indicate $95\%$ confidence intervals computed using stratified bootstrapping over five random seeds.
• Figure 4: Aggregated performance ranking of all the reward candidates on the MiniGrid (top) and Procgen (bottom) benchmarks. For simplicity, we abbreviate ICM, NGU, RE3, and E3B as I, N, R, and E. The mean and standard error are computed across all the environments.
• Figure 5: Cumulative distribution function of the performance from HIRE-1 to HIRE-4 on the MiniGrid (left) and Procgen (right) benchmarks.