Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method

TL;DR

Problem: sample-efficient learning in continuous control is hampered by inefficient replay sampling. Approach: RPE-PER uses EMCN to predict rewards and next-state transitions and prioritises experiences by Reward Prediction Error signals, integrated with TD3 and SAC. Contributions: (i) EMCN-based reward/transition modeling, (ii) RPE-driven prioritisation replacing standard TD-error-based scoring, and (iii) extensive MuJoCo evaluation showing faster learning and improved performance over state-of-the-art PER variants. Significance: the method offers a biologically inspired, model-based enhancement to off-policy RL that improves sample efficiency and practical performance in complex continuous-control tasks.

Abstract

Reinforcement Learning algorithms aim to learn optimal control strategies through iterative interactions with an environment. A critical element in this process is the experience replay buffer, which stores past experiences, allowing the algorithm to learn from a diverse range of interactions rather than just the most recent ones. This buffer is especially essential in dynamic environments with limited experiences. However, efficiently selecting high-value experiences to accelerate training remains a challenge. Drawing inspiration from the role of reward prediction errors (RPEs) in biological systems, where they are essential for adaptive behaviour and learning, we introduce Reward Predictive Error Prioritised Experience Replay (RPE-PER). This novel approach prioritises experiences in the buffer based on RPEs. Our method employs a critic network, EMCN, that predicts rewards in addition to the Q-values produced by standard critic networks. The discrepancy between these predicted and actual rewards is computed as RPE and utilised as a signal for experience prioritisation. Experimental evaluations across various continuous control tasks demonstrate RPE-PER's effectiveness in enhancing the learning speed and performance of off-policy actor-critic algorithms compared to baseline approaches.

Figures (4)

• Figure 1: A high-level representation of the RPE-PER framework and EMCN critic network. RPE is determined by computing the difference between the actual rewards stored in the buffer and the predicted rewards generated by EMCN, which is then used as a score for prioritisation experiences in the buffer.
• Figure 2: Six MuJoCo Tasks with Varying Complexity Levels Used in Our Experimental Evaluation
• Figure 3: Learning curves are generated in selected continuous control tasks in MuJoCo under TD3 policy method and using various sampling methods. Smoothing the curves for improved visual clarity is achieved by a sliding window of size 10. The solid lines represent the mean values, while the shaded regions indicate the standard deviations calculated from ten evaluations across ten runs with different random seeds.
• Figure 4: Learning curves are generated in selected continuous control tasks in MuJoCo under the SAC policy method using various sampling methods. Smoothing the curves for improved visual clarity is achieved by a sliding window of size 10. The solid lines represent the mean values, while the shaded regions indicate the standard deviations, calculated from ten evaluations across ten runs with different random seeds.