Investigating the Interplay of Prioritized Replay and Generalization

TL;DR

New insight is presented into the interaction between prioritization, bootstrapping, and neural networks and several improvements for Prioritized Experience Replay are proposed in tabular settings and noisy domains.

Abstract

Experience replay, the reuse of past data to improve sample efficiency, is ubiquitous in reinforcement learning. Though a variety of smart sampling schemes have been introduced to improve performance, uniform sampling by far remains the most common approach. One exception is Prioritized Experience Replay (PER), where sampling is done proportionally to TD errors, inspired by the success of prioritized sweeping in dynamic programming. The original work on PER showed improvements in Atari, but follow-up results were mixed. In this paper, we investigate several variations on PER, to attempt to understand where and when PER may be useful. Our findings in prediction tasks reveal that while PER can improve value propagation in tabular settings, behavior is significantly different when combined with neural networks. Certain mitigations $-$ like delaying target network updates to control generalization and using estimates of expected TD errors in PER to avoid chasing stochasticity $-$ can avoid large spikes in error with PER and neural networks but generally do not outperform uniform replay. In control tasks, none of the prioritized variants consistently outperform uniform replay. We present new insight into the interaction between prioritization, bootstrapping, and neural networks and propose several improvements for PER in tabular settings and noisy domains.

Figures (20)

• Figure 1: Prioritization can be problematic in noisy prediction with NNs. Results averaged over 30 trials; shaded region are 95% bootstrap Confidence Intervals (CI).
• Figure 2: The 50-state Markov chain environment.
• Figure 3: Prioritized methods can improve sample efficiency in prediction on the 50-state chain in tabular (left) and NN prediction (right). With NN function approximation Naive PER exhibits an increase in MSVE during early learning. The heatmaps show estimated values of the states, 1 to 50, over time. Results are averaged over 30 seeds; shaded regions are 95% bootstrap CI.
• Figure 4: Probability of sampling a transition starting from each state (1 to 50) from the buffer at each time point, in the 50-state Markov chain for one run.
• Figure 5: Target Networks can mitigate Naive PER's poor performance in the 50-state Markov chain prediction task with NNs. Red numbers above curves indicate Target Network update rate.