Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewards

Grant C. Forbes, Leonardo Villalobos-Arias, Jianxun Wang, Arnav Jhala, David L. Roberts

TL;DR

An extension to PBRS is presented that is proved preserves the set of optimal policies under a more general set of functions than has been previously proven, and it is proved that GRM is sufficiently general as to encompass all potential-based reward shaping functions.

Abstract

Recently there has been a proliferation of intrinsic motivation (IM) reward-shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present {\em Potential-Based Intrinsic Motivation} (PBIM) and {\em Generalized Reward Matching} (GRM), methods for converting IM rewards into a potential-based form that are useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff Walking environments, we demonstrate that PBIM and GRM successfully prevent the agent from converging to a suboptimal policy and can speed up training. Additionally, we prove that GRM is sufficiently general as to encompass all potential-based reward shaping functions. This paper expands on previous work introducing the PBIM method, and provides an extension to the more general method of GRM, as well as additional proofs, experimental results, and discussion.

Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewards

TL;DR

An extension to PBRS is presented that is proved preserves the set of optimal policies under a more general set of functions than has been previously proven, and it is proved that GRM is sufficiently general as to encompass all potential-based reward shaping functions.

Abstract

Recently there has been a proliferation of intrinsic motivation (IM) reward-shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present {\em Potential-Based Intrinsic Motivation} (PBIM) and {\em Generalized Reward Matching} (GRM), methods for converting IM rewards into a potential-based form that are useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff Walking environments, we demonstrate that PBIM and GRM successfully prevent the agent from converging to a suboptimal policy and can speed up training. Additionally, we prove that GRM is sufficiently general as to encompass all potential-based reward shaping functions. This paper expands on previous work introducing the PBIM method, and provides an extension to the more general method of GRM, as well as additional proofs, experimental results, and discussion.

Paper Structure

This paper contains 22 sections, 6 theorems, 51 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reward Shaping, Potential-Based And Otherwise
  4. Potential-Based Reward Shaping
  5. Intrinsic Motivation
  6. Extending Potential-Based Reward Shaping to Functions of Arbitrary Variables
  7. Converting Intrinsic Motivation to Potential-Based Reward Functions
  8. Generalized Reward Matching: A General Class of PBIM-Like Conversion Methods
  9. Introducing the Matching Function
  10. GRM Preserves Optimality
  11. PB-GRM and PBRS Are Equivalent
  12. A Representative Subset of Plausible GRM Functions
  13. Empirical Results
  14. Minigrid Doorkey
  15. Discussion: PBIM and GRM Outperform Baseline IM
  16. ...and 7 more sections

Key Result

Theorem 1

The addition of a shaping reward $F_t = \gamma \Phi_{t+1} - \Phi_t$ leaves the set of optimal policies unchanged if Equation boundary_condition holds.

Figures (10)

  • Figure 1: A comparison of our method with prior work in PBRS. A check here means that the domain or characteristic in question is treatable with or true of the method in question, while an 'X' indicates that it is not. The circle in the column for wiewiora2003potential indicates that optimality is preserved, but only if the resulting potential is added back to the Q table during policy training.
  • Figure 2: An example MiniGrid DoorKey 8x8 environment.
  • Figure 3: (\ref{['fig:025-full']}), (\ref{['frame_results_02']}), & (\ref{['frame_results_005']}) Frames per episode for each method (lower is better). The shaded region represents standard deviation, and plots are of a 100-point moving average.
  • Figure 4: Cliff walking scenario (adapted from sutton2018reinforcement).The red arrow shows the optimal path to take when no slip chance is present, while the blue arrow shows a "cautious," suboptimal path that the agent may take, particularly if it is trying to maximize some IM term in centivizing exploration.
  • Figure 5: Average cumulative extrinsic return and episode length for the cliff walking environment. Error bars are standard deviations over 10 runs.
  • ...and 5 more figures

Theorems & Definitions (10)

  • Theorem 1: Sufficient Condition For Optimality
  • proof
  • Theorem 2: PBIM Preserves Optimality
  • proof
  • Theorem 5.1
  • proof
  • Corollary 2.1
  • Theorem 5.2
  • proof
  • Corollary 2.2