Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?

TL;DR

This work tackles the inefficiency of on-policy RL for large language models by exploring off-policy RL with stale data. It identifies a prosperity-before-collapse phenomenon where stale data can be informative before instability arises and introduces M2PO, a second-moment constrained policy optimization that preserves informative updates while suppressing extreme outliers. By defining the batch second moment $\\hat{M}_2 = \frac{1}{N} \sum_{i=1}^{N} (\\log r_i)^2$ and proving a bound on distribution shift, the method enables stable off-policy training with stale data. Empirically, across six model scales and eight benchmarks, M2PO matches on-policy performance under heavy staleness (\\,s\\ge 256) and dramatically reduces token clipping from about 1.22% to 0.06%, demonstrating scalable RL for LLM reasoning.

Abstract

Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a prosperity-before-collapse phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce M2PO (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable off-policy training even with data stale by at least 256 model updates and matches on-policy performance.

Figures (12)

• Figure 1: Comparison of on-policy GRPO and off-policy training under a staleness of 256 model updates on Qwen-2.5-32B. Left: Standard GRPO suffers from degradation with stale rollouts, while removing the trust region (GRPO no TR) reveals a clear prosperity-before-collapse phenomenon. In contrast, M2PO achieves stable training and matches on-policy performance even under high staleness. Right: Token clipping ratio comparison shows that M2PO dramatically reduces clipping events compared to GRPO with the same staleness, while avoiding training collapse.
• Figure 2: Average accuracy for RL with stale data on Qwen2.5-Math-7B.
• Figure 3: Prosperity before Collapse. Training without a trust region (TR) ($\epsilon=\infty$) under stale data ($s=256$) initially achieves higher performance than clipped training, sometimes even matching the on-policy baseline ($s=0$). However, it eventually collapses due to uncontrolled variance.
• Figure 4: (a) Clipping ratio during training on the Qwen-2.5-Math-7B model. (b) Relationship between average token entropy and the distance between the importance sampling ratio and 1.
• Figure 5: Training reward on Qwen-2.5-32B.

Theorems & Definitions (3)

• Theorem 5.1: Bounding $\chi^2$ by $M_2$
• proof
• proof