PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization

TL;DR

The paper addresses sparse, delayed rewards in long-horizon reasoning under critic-free policy optimization by introducing PRPO, which fuses dense Process Reward Model feedback with sparse outcome rewards through semantic segmentation and distribution alignment. It defines token-level advantages via segment-wise normalization and a fused per-token advantage AF_t(τ) = z_{process,t}(τ) + β(τ), with the global alignment E[𝒫_{process}'] = E[𝒫_{process}] + E[𝒫_{outcome}]. Empirically, PRPO yields consistent improvements over GRPO and PRM-Avg on math reasoning benchmarks (e.g., MATH, AMC, AIME) using eight rollouts and no value network, while maintaining critic-free efficiency. The approach demonstrates that dense process-level supervision can be effectively integrated into critic-free optimization, enabling finer credit assignment and more reliable reasoning strategies in complex tasks.

Abstract

Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization.

Figures (4)

• Figure 1: Comparison between outcome-level and process-level reward in RLHF.
• Figure 2: Comparison between GRPO and PRPO.
• Figure 3: Training accuracy on MATH. PRPO steadily improves both baselines.
• Figure 4: Ablation on relative process-reward normalization.