Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rewarded Region Replay (R3) for Policy Learning with Discrete Action Space

Bangzheng Li, Ningshan Ma, Zifan Wang

TL;DR

The paper addresses learning under sparse rewards in discrete-action RL by proposing Rewarded Region Replay (R3), an on-policy-PPO framework augmented with a replay buffer of past successful trajectories and capped importance sampling to control variance. The core idea blends on-policy stability with off-policy sample efficiency, enabling faster and more reliable learning in challenging environments. Results on Crossing, DoorKey, and CartPole demonstrate that R3 and its dense reward extension DR3 outperform PPO, with DR3 showing strong performance in dense reward settings and R3 providing clear benefits as task complexity increases. The work highlights practical benefits for sample-efficient policy learning in discrete action spaces and provides code for reproducibility at the given repository.

Abstract

We introduce a new on-policy algorithm called Rewarded Region Replay (R3), which significantly improves on PPO in solving environments with discrete action spaces. R3 improves sample efficiency by using a replay buffer which contains past successful trajectories with reward above a certain threshold, which are used to update a PPO agent with importance sampling. Crucially, we discard the importance sampling factors which are above a certain ratio to reduce variance and stabilize training. We found that R3 significantly outperforms PPO in Minigrid environments with sparse rewards and discrete action space, such as DoorKeyEnv and CrossingEnv, and moreover we found that the improvement margin of our method versus baseline PPO increases with the complexity of the environment. We also benchmarked the performance of R3 against DDQN (Double Deep Q-Network), which is a standard baseline in off-policy methods for discrete actions, and found that R3 also outperforms DDQN agent in DoorKeyEnv. Lastly, we adapt the idea of R3 to dense reward setting to obtain the Dense R3 algorithm (or DR3) and benchmarked it against PPO on Cartpole-V1 environment. We found that DR3 outperforms PPO significantly on this dense reward environment. Our code can be found at https://github.com/chry-santhemum/R3.

Rewarded Region Replay (R3) for Policy Learning with Discrete Action Space

TL;DR

The paper addresses learning under sparse rewards in discrete-action RL by proposing Rewarded Region Replay (R3), an on-policy-PPO framework augmented with a replay buffer of past successful trajectories and capped importance sampling to control variance. The core idea blends on-policy stability with off-policy sample efficiency, enabling faster and more reliable learning in challenging environments. Results on Crossing, DoorKey, and CartPole demonstrate that R3 and its dense reward extension DR3 outperform PPO, with DR3 showing strong performance in dense reward settings and R3 providing clear benefits as task complexity increases. The work highlights practical benefits for sample-efficient policy learning in discrete action spaces and provides code for reproducibility at the given repository.

Abstract

We introduce a new on-policy algorithm called Rewarded Region Replay (R3), which significantly improves on PPO in solving environments with discrete action spaces. R3 improves sample efficiency by using a replay buffer which contains past successful trajectories with reward above a certain threshold, which are used to update a PPO agent with importance sampling. Crucially, we discard the importance sampling factors which are above a certain ratio to reduce variance and stabilize training. We found that R3 significantly outperforms PPO in Minigrid environments with sparse rewards and discrete action space, such as DoorKeyEnv and CrossingEnv, and moreover we found that the improvement margin of our method versus baseline PPO increases with the complexity of the environment. We also benchmarked the performance of R3 against DDQN (Double Deep Q-Network), which is a standard baseline in off-policy methods for discrete actions, and found that R3 also outperforms DDQN agent in DoorKeyEnv. Lastly, we adapt the idea of R3 to dense reward setting to obtain the Dense R3 algorithm (or DR3) and benchmarked it against PPO on Cartpole-V1 environment. We found that DR3 outperforms PPO significantly on this dense reward environment. Our code can be found at https://github.com/chry-santhemum/R3.
Paper Structure (19 sections, 7 figures, 3 tables, 3 algorithms)

This paper contains 19 sections, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proximal policy optimization (PPO)
  4. Double deep Q-network (DDQN)
  5. Experience replay
  6. Methods
  7. Environment
  8. Weak R3
  9. R3
  10. DR3
  11. Results
  12. Results in Crossing Env
  13. Results in DoorKey Env
  14. Results in CartPole Env
  15. Future work
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Flowchart of R3
  • Figure 2: Performance of R3, PPO, and DDQN agent on Crossing environment
  • Figure 3: Performance of R3 vs PPO agent on Crossing environment
  • Figure 4: Performance of R3, PPO, and DDQN agent on DoorKey environment
  • Figure 5: Performance of R3 vs PPO agent on DoorKey environment
  • ...and 2 more figures