Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen, Tianyi Lin
TL;DR
This work tackles the all-negative-sample problem in group relative policy optimization (GRPO) for LLM reasoning by introducing Stepwise Guided Policy Optimization (SGPO). SGPO uses a step-wise judge to identify the first incorrect step in a reasoning trajectory, converting negative samples into informative step-level rewards via a reasoning trajectory score (RTS) and a calibrated reward r_SGPO, enabling learning signals even when all samples fail. The authors provide theoretical analysis in a simplified two-step setting showing SGPO accelerates learning and converges to the optimal policy, and they validate the approach empirically across 7B, 14B, and 32B models in offline and online RL on nine benchmarks, using a range of judge models from open-source to proprietary. Importantly, SGPO does not require judges to produce correct answers, improving practicality and accessibility, and demonstrates robustness across resource settings while enhancing early- and mid-training gains where all-negative groups are prevalent.
Abstract
Reinforcement learning (RL) has proven effective in strengthening the reasoning capabilities of large language models (LLMs). A widely adopted method, Group Relative Policy Optimization (GRPO), has shown strong empirical results in training DeepSeek-R1. However, GRPO fails to update the policy when all responses within a group are incorrect (i.e., \emph{all-negative-sample} groups). This limitation underscores a key gap between artificial and human intelligence: unlike humans, who can learn from mistakes, GRPO discards these signals. Our first contribution is to introduce a simple framework that mitigates the all-negative-sample issue by incorporating response diversity within groups using a \textit{step-wise} judge model, which can be either directly trained or adapted from existing LLMs. We prove that this diversification can accelerate GRPO's learning dynamics in a simplified setting. We also empirically validate the proposed stepwise guided policy optimization (SGPO) method, demonstrating consistent gains across model sizes (7B, 14B, 32B) in offline and online training on 9 benchmarks, including base and distilled variants. Our results highlight two advantages: (i) SGPO surpasses GRPO, especially in the early and mid-training stages where all-negative-sample groups are prevalent; and (ii) SGPO does not require judge models to generate correct answers, differentiating it from knowledge distillation methods.