Action-Dependent Optimality-Preserving Reward Shaping
Grant C. Forbes, Jianxun Wang, Leonardo Villalobos-Arias, Arnav Jhala, David L. Roberts
TL;DR
The paper addresses the challenge of using intrinsic motivation (IM) for exploration without altering the optimal policy in sparse-reward RL. It critiques existing PBRS-based approaches (PBIM, GRM, PIES) for failing in long-horizon, exploration-heavy environments such as Montezuma's Revenge, and introduces Action-Dependent Optimality-Preserving Shaping (ADOPS) as a general, critic-informed method to convert IM into an optimality-preserving form. It provides both ideal and practical formulations, proving that ADOPS preserves the optimal policy set even when shaping rewards depend on actions, and demonstrates empirical superiority over PBIM, GRM, PIES, and baselines, achieving state-of-the-art results in the target domain. The key contributions include a generalized optimality condition beyond PBRS, a provable ADOPS framework with critic-based approximations, and extensive experiments showing robust improvement in extremely sparse environments. The practical impact lies in enabling reliable use of IM to accelerate learning in complex, real-world tasks where long episodes and sparse rewards previously hindered policy optimization.
Abstract
Recent RL research has utilized reward shaping--particularly complex shaping rewards such as intrinsic motivation (IM)--to encourage agent exploration in sparse-reward environments. While often effective, ``reward hacking'' can lead to the shaping reward being optimized at the expense of the extrinsic reward, resulting in a suboptimal policy. Potential-Based Reward Shaping (PBRS) techniques such as Generalized Reward Matching (GRM) and Policy-Invariant Explicit Shaping (PIES) have mitigated this. These methods allow for implementing IM without altering optimal policies. In this work we show that they are effectively unsuitable for complex, exploration-heavy environments with long-duration episodes. To remedy this, we introduce Action-Dependent Optimality Preserving Shaping (ADOPS), a method of converting intrinsic rewards to an optimality-preserving form that allows agents to utilize IM more effectively in the extremely sparse environment of Montezuma's Revenge. We also prove ADOPS accommodates reward shaping functions that cannot be written in a potential-based form: while PBRS-based methods require the cumulative discounted intrinsic return be independent of actions, ADOPS allows for intrinsic cumulative returns to be dependent on agents' actions while still preserving the optimal policy set. We show how action-dependence enables ADOPS's to preserve optimality while learning in complex, sparse-reward environments where other methods struggle.