Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization

Shiye Lei, Zhihao Cheng, Dacheng Tao

TL;DR

This work identifies instability in off-policy RL for LLM post-training as stemming from token-level importance sampling, showing that the theoretically correct correction is the prefix importance ratio $\rho_{1:t}$. It introduces MinPRO, a stable surrogate that multiplies the token-level ratio $\rho_t$ by the minimum prior prefix ratio $\underline{\rho}_t$, avoiding the variance and length-bias issues of cumulative products. The method yields significantly more stable training and higher peak performance across dense and mixture-of-experts LLMs on multiple mathematical reasoning benchmarks, under large off-policy drift. The results demonstrate that prefix-aware corrections generalize to MoE scales and offer a practical, scalable route to improved reasoning in LLMs. Overall, MinPRO provides a principled stabilization mechanism that addresses a core bottleneck in off-policy LLM policy optimization with clear implications for future large-scale training.

Abstract

Reinforcement learning (RL) post-training has increasingly demonstrated strong ability to elicit reasoning behaviors in large language models (LLMs). For training efficiency, rollouts are typically generated in an off-policy manner using an older sampling policy and then used to update the current target policy. To correct the resulting discrepancy between the sampling and target policies, most existing RL objectives rely on a token-level importance sampling ratio, primarily due to its computational simplicity and numerical stability. However, we observe that token-level correction often leads to unstable training dynamics when the degree of off-policyness is large. In this paper, we revisit LLM policy optimization under off-policy conditions and show that the theoretically rigorous correction term is the prefix importance ratio, and that relaxing it to a token-level approximation can induce instability in RL post-training. To stabilize LLM optimization under large off-policy drift, we propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO). MinPRO replaces the unstable cumulative prefix ratio with a non-cumulative surrogate based on the minimum token-level ratio observed in the preceding prefix. Extensive experiments on both dense and mixture-of-experts LLMs, across multiple mathematical reasoning benchmarks, demonstrate that MinPRO substantially improves training stability and peak performance in off-policy regimes.

A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization

TL;DR

This work identifies instability in off-policy RL for LLM post-training as stemming from token-level importance sampling, showing that the theoretically correct correction is the prefix importance ratio . It introduces MinPRO, a stable surrogate that multiplies the token-level ratio by the minimum prior prefix ratio , avoiding the variance and length-bias issues of cumulative products. The method yields significantly more stable training and higher peak performance across dense and mixture-of-experts LLMs on multiple mathematical reasoning benchmarks, under large off-policy drift. The results demonstrate that prefix-aware corrections generalize to MoE scales and offer a practical, scalable route to improved reasoning in LLMs. Overall, MinPRO provides a principled stabilization mechanism that addresses a core bottleneck in off-policy LLM policy optimization with clear implications for future large-scale training.

Abstract

Reinforcement learning (RL) post-training has increasingly demonstrated strong ability to elicit reasoning behaviors in large language models (LLMs). For training efficiency, rollouts are typically generated in an off-policy manner using an older sampling policy and then used to update the current target policy. To correct the resulting discrepancy between the sampling and target policies, most existing RL objectives rely on a token-level importance sampling ratio, primarily due to its computational simplicity and numerical stability. However, we observe that token-level correction often leads to unstable training dynamics when the degree of off-policyness is large. In this paper, we revisit LLM policy optimization under off-policy conditions and show that the theoretically rigorous correction term is the prefix importance ratio, and that relaxing it to a token-level approximation can induce instability in RL post-training. To stabilize LLM optimization under large off-policy drift, we propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO). MinPRO replaces the unstable cumulative prefix ratio with a non-cumulative surrogate based on the minimum token-level ratio observed in the preceding prefix. Extensive experiments on both dense and mixture-of-experts LLMs, across multiple mathematical reasoning benchmarks, demonstrate that MinPRO substantially improves training stability and peak performance in off-policy regimes.
Paper Structure (18 sections, 3 theorems, 20 equations, 6 figures, 4 tables)

This paper contains 18 sections, 3 theorems, 20 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM Policy Optimization
  4. Off-Policy RL in LLM
  5. Preliminaries
  6. Method
  7. A Step Back
  8. MinPRO
  9. Experiments
  10. Main Results
  11. Scaling to Large MoE LLMs
  12. Discussion
  13. Post-hoc Analysis
  14. Unsuccessful Attempts
  15. Conclusion
  16. ...and 3 more sections

Key Result

Lemma 1

Let $\bm{o}= (o_1, o_2, \ldots, o_T)$ denote a trajectory generated by $\pi_\theta$. The gradient of the RL objective satisfies where $\bm{o}_{< t} = (o_1,..., o_{t-1})$ is the prefix prior to step $t$ and $A^\pi(o_t;\bm{o}_{<t})$ is the advantage under $\pi_\theta$.

Figures (6)

  • Figure 1: (a) An overview of MinPRO. $\nabla_\theta \mathcal{J}$ denotes the policy gradient and $\rho_t$ is the token-level importance sampling (IS) ratio. We take a step back to derive the full IS ratio $\rho_{1:t}$, referred to as the prefix importance ratio. MinPRO is then developed by relaxing $\rho_{1:t} = \rho_{1:t-1} \cdot \rho_t$ to a non-cumulative proxy $\underline{\rho}_t \rho_t$. (b) AIME24 and AIME25 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy training.
  • Figure 2: Plots of (a) reward and (b) entropy as functions of training steps for Qwen3-8B-Base under on-policy and off-policy regimes. In the off-policy setting, each batch of sampled rollouts is stored in a buffer and used for training after a delay of $2$ global steps, whereas the on-policy setting applies no delay between sampling and optimization.
  • Figure 3: Training reward curves as functions of training steps for Qwen3-8B-Base and Qwen3-14B-Base under off-policy optimization.
  • Figure 4: (a) Training reward curves and (b) average pass@1 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy optimization.
  • Figure 5: Hard-clipped token fraction as a function of training steps for GRPO and M2PO under off-policy optimization.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Lemma 1: Policy Gradient Theorem sutton1999policy
  • Theorem 1: Policy Gradient under Off-policy Conditions
  • Lemma 2: Score-function baseline invariance
  • proof