Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning

TL;DR

This work addresses offline RL data efficiency by introducing trajectory-based replay concepts: Trajectory Replay (TR) stores data as complete trajectories and samples them backward to leverage future-state information, while Prioritized Trajectory Replay (PTR) adds trajectory-level priorities based on quality and uncertainty to further improve learning. A SARSA-style weighted target is proposed to limit extrapolation errors when updating critics, yielding a weighted blend between the original target and the SARSA target. Empirical results on D4RL show that TR improves performance on sparse-reward tasks and that PTR’s trajectory-quality or uncertainty-based prioritization can offer robust gains across diverse environments, with some limitations in dense-reward settings and extremely sparse datasets. Overall, the paper highlights trajectory-centric data sampling as a promising, plug-and-play enhancement for data-driven reinforcement learning, while outlining directions for theoretical analysis and data-collection guidance in practice.

Abstract

In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL. In summary, our research emphasizes the significance of trajectory-based data sampling techniques in enhancing the efficiency and performance of offline RL algorithms.

Figures (4)

• Figure 1: A motivating example on finite data. Left: The illustration of state transitions in three trajectories $\tau_i$ started at state $s_0$, with the reward $r$ for labeled state or 0 for others, and four different sampling techniques. Middle and right: Curves of the estimated maximum $Q$-value at state $s_0$ learned on these three trajectories. The solid line is the averaged value over 50 seeds, and the shaded area the standard variance. The oracle value, taking into account the discount factor, is slightly less than 8.
• Figure 2: Overview of the process of data sampling based on Trajectory Replay.
• Figure 3: Comparison of the performance of PTR compared to TD3+BC under 10 different trajectory priority metrics. In the figure we restrict the performance difference to a maximum of 20.
• Figure 4: The motivating example on finite data.