Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intrinsic Reward Policy Optimization for Sparse-Reward Environments

Minjae Cho, Huy Trong Tran

TL;DR

This work addresses the challenge of sparse rewards in reinforcement learning by introducing Intrinsic Reward Policy Optimization (IRPO), a framework that uses multiple intrinsic rewards to train exploratory policies and backpropagate their extrinsic signals to update a single base policy. By constructing an IRPO gradient and leveraging a bi-level actor-critic structure with Jacobian-enabled backpropagation, IRPO directly optimizes for extrinsic rewards without pretraining subpolicies. Empirical results show IRPO improves performance and sample efficiency across discrete and continuous tasks, and a formal analysis outlines the optimization dynamics and potential optimality under certain conditions. The approach offers a principled, scalable alternative to standard exploration methods and hierarchical RL for sparse-reward environments.

Abstract

Exploration is essential in reinforcement learning as an agent relies on trial and error to learn an optimal policy. However, when rewards are sparse, naive exploration strategies, like noise injection, are often insufficient. Intrinsic rewards can also provide principled guidance for exploration by, for example, combining them with extrinsic rewards to optimize a policy or using them to train subpolicies for hierarchical learning. However, the former approach suffers from unstable credit assignment, while the latter exhibits sample inefficiency and sub-optimality. We propose a policy optimization framework that leverages multiple intrinsic rewards to directly optimize a policy for an extrinsic reward without pretraining subpolicies. Our algorithm -- intrinsic reward policy optimization (IRPO) -- achieves this by using a surrogate policy gradient that provides a more informative learning signal than the true gradient in sparse-reward environments. We demonstrate that IRPO improves performance and sample efficiency relative to baselines in discrete and continuous environments, and formally analyze the optimization problem solved by IRPO. Our code is available at https://github.com/Mgineer117/IRPO.

Intrinsic Reward Policy Optimization for Sparse-Reward Environments

TL;DR

This work addresses the challenge of sparse rewards in reinforcement learning by introducing Intrinsic Reward Policy Optimization (IRPO), a framework that uses multiple intrinsic rewards to train exploratory policies and backpropagate their extrinsic signals to update a single base policy. By constructing an IRPO gradient and leveraging a bi-level actor-critic structure with Jacobian-enabled backpropagation, IRPO directly optimizes for extrinsic rewards without pretraining subpolicies. Empirical results show IRPO improves performance and sample efficiency across discrete and continuous tasks, and a formal analysis outlines the optimization dynamics and potential optimality under certain conditions. The approach offers a principled, scalable alternative to standard exploration methods and hierarchical RL for sparse-reward environments.

Abstract

Exploration is essential in reinforcement learning as an agent relies on trial and error to learn an optimal policy. However, when rewards are sparse, naive exploration strategies, like noise injection, are often insufficient. Intrinsic rewards can also provide principled guidance for exploration by, for example, combining them with extrinsic rewards to optimize a policy or using them to train subpolicies for hierarchical learning. However, the former approach suffers from unstable credit assignment, while the latter exhibits sample inefficiency and sub-optimality. We propose a policy optimization framework that leverages multiple intrinsic rewards to directly optimize a policy for an extrinsic reward without pretraining subpolicies. Our algorithm -- intrinsic reward policy optimization (IRPO) -- achieves this by using a surrogate policy gradient that provides a more informative learning signal than the true gradient in sparse-reward environments. We demonstrate that IRPO improves performance and sample efficiency relative to baselines in discrete and continuous environments, and formally analyze the optimization problem solved by IRPO. Our code is available at https://github.com/Mgineer117/IRPO.
Paper Structure (29 sections, 3 theorems, 32 equations, 9 figures, 2 tables, 1 algorithm)

This paper contains 29 sections, 3 theorems, 32 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Our Algorithm: Intrinsic Reward Policy Optimization
  4. Exploratory Policy Updates
  5. Updating exploratory policies.
  6. Storing Jacobians for future base policy updates.
  7. Base Policy Updates
  8. IRPO gradient.
  9. Estimating the IRPO gradient with backpropagation.
  10. Trust-region update.
  11. Formal Analysis of IRPO
  12. Empirical Analysis of IRPO
  13. Experiments
  14. Main Results
  15. Ablation Study
  16. ...and 14 more sections

Key Result

Corollary 3.1

Assume a sparse-reward setting as in Assumption assump:sparse_rewards, a bounded policy log-gradient as in Assumption assump:bound_on_the_policy_log_gradient, and a discount factor $\gamma \in [0, 1)$. Then, the $\ell_2$-norm of the policy gradient of any stochastic policy $\pi_\theta$ approaches ze

Figures (9)

  • Figure 1: (a) IRPO optimizes multiple exploratory policies (from the current base policy $\pi_{\theta^{(i)}}$) using intrinsic rewards and uses their gradients to update the base policy via backpropagation. (b) Policy noise injection methods generate nearby policies by adding random noise and use their gradients to update the base policy. (c) Uncertainty-based methods augment the true (extrinsic) reward with intrinsic rewards based on various forms of uncertainty. (d) Hierarchical RL pretrains subpolicies (based on intrinsic rewards) and uses them as temporally extended actions. Unlike the other methods, our approach is less restricted in its search range, directly optimizes for extrinsic rewards, and does not require pretrained subpolicies.
  • Figure 2: We assume an actor-critic framework with multiple exploratory policies, each having its own intrinsic and extrinsic critics, which are used to update the exploratory and base policy, respectively.
  • Figure 3: We show the update mechanism of IRPO with a single intrinsic reward (i.e., $K = 1$). We plot the results for different numbers of exploratory policy updates, $N=2$ and $N=5$, respectively, with extrinsic performance plotted on a contour.
  • Figure 4: The environments used in our experiments. We consider three discrete environments (top row) and six continuous ones (middle and bottom rows).
  • Figure 5: Learning curves for environments in \ref{['fig:env']}, with mean and 95% confidence intervals over 10 random seeds. Learning curves for IRPO-ALLO (ours) and HRL-ALLO include samples needed to derive intrinsic rewards (both) and to pretrain subpolicies (HRL-ALLO only). The locomotion threshold in AntMaze environments refers to the stage where the agent starts learning to walk.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Corollary 3.1: Vanishing Policy Gradient in Sparse-Reward Settings
  • Definition 3.2: Reachable Exploratory Policies
  • Definition 3.3: All Reachable Exploratory Policies
  • Remark 3.4: Optimization Problem of IRPO
  • Lemma 1.3: Bounded Expected Action-Value Function in Sparse-Reward Settings
  • proof
  • Lemma 1.4: Bounded Policy Gradient in Sparse-Reward Settings
  • proof
  • proof