Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning

TL;DR

Massively parallel RL with short PPO rollouts suffers from cyclical nonstationarity caused by synchronous resets, which biases batch data and destabilizes learning. The authors introduce staggered resets to initialize parallel environments at diverse effective times within the task horizon, creating temporally diverse batches without altering the learning algorithm. Across toy environments and high-dimensional ManiSkill3 and AllegroKuka tasks, staggered resets yield higher sample efficiency, faster wall-clock convergence, and stronger final performance, while scaling better with more parallel environments. The approach is algorithm-agnostic, improving stability of value estimates and reducing forgetting, thereby enabling more effective learning in long-horizon robotic tasks on GPU-accelerated simulators.

Abstract

Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common to use short rollouts per policy update, increasing the update-to-data (UTD) ra- tio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset at varied points within the task horizon. This yields training batches with greater temporal diversity, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through illustrative toy environ- ments. We then apply this technique to challenging high-dimensional robotics environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts.

Figures (13)

• Figure 1: Data collection in massively parallel RL. Rows are environments, columns are time within task horizon $H$. Colors (red, orange, etc.) mark distinct task stages. (a) Synchronous Resets (Naive): All environments start at $t=0$ (red stage). Each rollout batch (e.g., Rollout 1: all red; Rollout 2: all orange) is temporally homogeneous. Batch content cycles through stages every $H/K$ rollouts, causing cyclical nonstationarity for the learner. (b) Staggered Resets: Environments start and hence at varied points in the task. Each rollout batch contains a mix of task stages (red, orange, green, blue, purple). This within-batch temporal diversity is maintained across rollouts, yielding a more stationary and representative data distribution.
• Figure 2: PPO with Non-Staggered (blue) vs. Staggered (red) resets on toy environments (mean $\pm$ 1 std dev). Staggered resets show robust performance as (a) horizon $H$ increases, (b) reset homogeneity ($2 -\lambda_R$) increases, and (c) progression probability ($p_{\text{prog}}$) varies, unlike non-staggered PPO which degrades especially with longer horizons, more deterministic resets, and easier skill gates ($p_{\text{prog}} > 0$).
• Figure 3: State visitation KDEs in StackCube-v1 over five rollouts. (a) Long Rollout ($K=100$): stable, broad coverage. (b) Naive Short Rollout ($K=25$): cyclical non-stationarity, narrow/erratic coverage. (c) Staggered Short Rollout ($K=25$): stable, diverse coverage, emulating (a) despite short trajectories.
• Figure 4: Staggered Resets (Ours, red) consistently outperform Synchronous Resets (blue) across different on-policy algorithms and task suites. Plots show the average evaluation metric (success rate or reward) vs. environment steps. Shaded areas show the standard deviation over 10 seeds. (Top & Middle Rows) On PPO with diverse ManiSkill3 tasks (StackCube, PushT, etc.), staggered resets consistently improve learning speed, final performance, and stability. Performance is comparable only on the locomotion task MS-HumanoidWalk, where natural desynchronization reduces the severity of the problem. (Bottom Row) To demonstrate algorithm-agnosticism, we evaluate on SAPG with challenging AllegroKuka manipulation tasks. Staggered resets again yield substantial gains in sample efficiency and final reward, confirming the generality of our approach.
• Figure 5: Wall-clock time to convergence versus number of parallel environments ($N$) for (a) StackCube-v1 and (b) Unitree G1 Transport Box