Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning

TL;DR

This work tackles the sparse-reward credit assignment problem in RL for complex LLM reasoning by introducing Verifiable Prefix Policy Optimization (VPPO). VPPO uses a fixed Process Reward Model (PRM) to locate the first incorrect step in a reasoning trajectory, preserves the correct prefix up to that point, and applies an additional reward to tokens within a reward prefix, while penalizing only the erroneous suffix. A shorten-prefix strategy is proposed to avoid artificial inflation of reasoning steps, and a RELU variant is explored for high-capability models to stabilize learning. Empirically, VPPO with shorten-prefix outperforms GRPO and prior PRM-guided approaches across a range of benchmarks (e.g., AIME, MATH, Minerva, HMMT) in both Pass@1 and Pass@K, demonstrating improved credit assignment, exploration, and learning signal quality for LLM reasoning.

Abstract

Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs). However, most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. Process reward models (PRMs) offer fine-grained step-level supervision, but their scores are often noisy and difficult to evaluate. As a result, recent PRM benchmarks focus on a more objective capability: detecting the first incorrect step in a reasoning path. However, this evaluation target is misaligned with how PRMs are typically used in RL, where their step-wise scores are treated as raw rewards to maximize. To bridge this gap, we propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL. Given an incorrect rollout, VPPO partitions the trajectory into a verified correct prefix and an erroneous suffix based on the first error, rewarding the former while applying targeted penalties only after the detected mistake. This design yields stable, interpretable learning signals and improves credit assignment. Across multiple reasoning benchmarks, VPPO consistently outperforms sparse-reward RL and prior PRM-guided baselines on both Pass@1 and Pass@K.

Figures (6)

• Figure 1: Distribution of Correct Steps before the First Error in Incorrect Responses. Among incorrect trajectories, 88% contain at least one correct step before the first error. We define the correct-step ratio as the number of steps before the first incorrect step divided by the total number of steps; the average correct-step ratio is 34%.
• Figure 2: Exploration Example Comparison. When two rollouts have different prefix, sparse reward only encourages the sampled correct path but our reward scheme enhances the likelihood for more correct paths.
• Figure 3: Exploitation Example Comparison. When two rollouts have similar prefix but different answers, sparse reward have conflict signal on the shared prefix but our reward scheme can accurately enhance the correct sample.
• Figure 4: Training Dynamics of Test Accuracy on AIME2025. Our method outperform all baselines during training.
• Figure 5: Training Steps and Reasoning Step Number. The x-axis shows the training step, and the y-axis reports the average number of reasoning steps in responses generated on AIME25, where we sample 16 responses per question at each training step. Rewarding the simple prefix substantially increases the reasoning step number compared with rewarding the shortened prefix.

Theorems & Definitions (2)

• Theorem 1: Informal
• proof