Selective Experience Replay for Lifelong Learning

TL;DR

The paper tackles catastrophic forgetting in continual reinforcement learning by introducing selective experience replay, augmenting the standard memory with a long-term store and evaluating four targeted experience-selection strategies. It finds distribution matching to be the most effective and robust for preventing forgetting across domains, while coverage-based selection helps when certain tasks are disproportionately important. By integrating established RL techniques (Q-learning, n-step returns, dynamic frame skipping) with two state-action representations and a replay-buffer framework, the study demonstrates improved retention and learning efficiency in autonomous-vehicle intersection tasks. These findings highlight the practical potential of selective replay for real-world, multi-task RL systems, while also outlining directions for broader, more scalable lifelong learning.

Abstract

Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial - when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.

Figures (9)

• Figure 1: Crossing a busy intersection. Red car is the autonomous vehicle and yellow cars are oncoming traffic. The objective is to determine the acceleration profile along the path.
• Figure 2: Visualizations of different intersection scenarios.
• Figure 3: Fine-tuning comparison.
• Figure 4: Comparison of results for all methods and scenarios.
• Figure 5: Trade-off between the time to cross and collision rate as the TTC threshold is varied. Note that performance of the DQN dominates in every case. The challenge scenario is excluded for scale reasons, but the results are similar.