Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiency

TL;DR

SnapshotRL addresses DRL sample inefficiency by altering environments rather than changing algorithms, leveraging complete environment snapshots from teacher trajectories. The framework standardizes snapshot collection and introduces a resetting approach that uses snapshots only in the early training phase, followed by evaluation in the original environment. The S3RL baseline adds Status Classification and Student Trajectory Truncation to maximize the influence of snapshots, yielding notable gains in sample efficiency and average return when paired with TD3 and SAC on MuJoCo, with more limited improvements for PPO. This approach enables efficient reuse of prior computational work without extra data or compute, offering a flexible, scalable path for accelerating DRL research and deployment.

Abstract

Deep reinforcement learning (DRL) algorithms require substantial samples and computational resources to achieve higher performance, which restricts their practical application and poses challenges for further development. Given the constraint of limited resources, it is essential to leverage existing computational work (e.g., learned policies, samples) to enhance sample efficiency and reduce the computational resource consumption of DRL algorithms. Previous works to leverage existing computational work require intrusive modifications to existing algorithms and models, designed specifically for specific algorithms, lacking flexibility and universality. In this paper, we present the Snapshot Reinforcement Learning (SnapshotRL) framework, which enhances sample efficiency by simply altering environments, without making any modifications to algorithms and models. By allowing student agents to choose states in teacher trajectories as the initial state to sample, SnapshotRL can effectively utilize teacher trajectories to assist student agents in training, allowing student agents to explore a larger state space at the early training phase. We propose a simple and effective SnapshotRL baseline algorithm, S3RL, which integrates well with existing DRL algorithms. Our experiments demonstrate that integrating S3RL with TD3, SAC, and PPO algorithms on the MuJoCo benchmark significantly improves sample efficiency and average return, without extra samples and additional computational resources.

Figures (15)

• Figure 1: Schematic of S3RL training process. The figure illustrates a teacher trajectory (light blue line with outline) from the initial point (red pin) to the goal point (yellow pentagram). Dark blue dots scattered on this trajectory indicate environment snapshots obtained from the teacher agent's interaction with environment, from which the student agent starts new training represented by green trajectories. Truncation points(black and yellow squares) on the right of three student trajectories signify truncated training implemented to prevent the student agent from deviating excessively from the teacher trajectory. The student trajectory on far right reaches the goal point, demonstrating that the student agent can successfully accomplish tasks. The figure vividly portrays the mechanism and objective of S3RL: to support the training of new agents effectively by leveraging environment snapshots.
• Figure 2: Learning curves sample efficiency comparison of TD3, SnapshotRL+TD3, and S3RL+TD3 on six MuJoCo environments. For individual environment results, see Figure \ref{['fig:td3_indiv']}.
• Figure 3: Ablation study results showing the impact of key components on the sample efficiency of S3RL+TD3 on six MuJoCo environments. For individual environment results, see Figure \ref{['fig:td3_ablation_indiv']}.
• Figure 4: Learning curves sample efficiency sweeps for S3RL+TD3 across $K$ on six MuJoCo environments. For individual environment results, see Figure \ref{['fig:td3_sweep_k_indiv']}.
• Figure 5: Learning curves sample efficiency sweeps for S3RL+TD3 across $T$ on six MuJoCo environments. For individual environment results, see Figure \ref{['fig:td3_sweep_t_indiv']}.