Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRL: Process Reward Learning Improves LLMs' Reasoning Ability and Broadens the Reasoning Boundary

Jiarui Yao, Ruida Wang, Tong Zhang

TL;DR

This work introduces Process Reward Learning (PRL), a theoretically grounded framework that turns sparse outcome rewards into dense, intermediate supervision by decomposing the entropy-regularized RL objective into intermediate steps. PRL derives an optimal process reward that leads to a natural policy update where $\pi^*(a|x) \propto \pi_0(a|x) e^{\eta r^*(x,a)}$, enabling credit assignment to intermediate reasoning steps without resorting to costly MCTS or separate reward models. Empirically, PRL improves average reasoning performance and broadens the reasoning boundary on standard math-reasoning benchmarks across multiple base models, demonstrating both higher accuracy and greater problem coverage. The approach provides a rigorous, efficient link between intermediate process signals and the global optimization objective, with potential for scalable reasoning improvements in large language models.

Abstract

Improving the reasoning abilities of Large Language Models (LLMs) has been a continuous topic recently. But most relevant works are based on outcome rewards at the trajectory level, missing fine-grained supervision during the reasoning process. Other existing training frameworks that try to combine process signals together to optimize LLMs also rely heavily on tedious additional steps like MCTS, training a separate reward model, etc., doing harm to the training efficiency. Moreover, the intuition behind the process signals design lacks rigorous theoretical support, leaving the understanding of the optimization mechanism opaque. In this paper, we propose Process Reward Learning (PRL), which decomposes the entropy regularized reinforcement learning objective into intermediate steps, with rigorous process rewards that could be assigned to models accordingly. Starting from theoretical motivation, we derive the formulation of PRL that is essentially equivalent to the objective of reward maximization plus a KL-divergence penalty term between the policy model and a reference model. However, PRL could turn the outcome reward into process supervision signals, which helps better guide the exploration during RL optimization. From our experiment results, we demonstrate that PRL not only improves the average performance for LLMs' reasoning ability measured by average @ n, but also broadens the reasoning boundary by improving the pass @ n metric. Extensive experiments show the effectiveness of PRL could be verified and generalized.

PRL: Process Reward Learning Improves LLMs' Reasoning Ability and Broadens the Reasoning Boundary

TL;DR

This work introduces Process Reward Learning (PRL), a theoretically grounded framework that turns sparse outcome rewards into dense, intermediate supervision by decomposing the entropy-regularized RL objective into intermediate steps. PRL derives an optimal process reward that leads to a natural policy update where , enabling credit assignment to intermediate reasoning steps without resorting to costly MCTS or separate reward models. Empirically, PRL improves average reasoning performance and broadens the reasoning boundary on standard math-reasoning benchmarks across multiple base models, demonstrating both higher accuracy and greater problem coverage. The approach provides a rigorous, efficient link between intermediate process signals and the global optimization objective, with potential for scalable reasoning improvements in large language models.

Abstract

Improving the reasoning abilities of Large Language Models (LLMs) has been a continuous topic recently. But most relevant works are based on outcome rewards at the trajectory level, missing fine-grained supervision during the reasoning process. Other existing training frameworks that try to combine process signals together to optimize LLMs also rely heavily on tedious additional steps like MCTS, training a separate reward model, etc., doing harm to the training efficiency. Moreover, the intuition behind the process signals design lacks rigorous theoretical support, leaving the understanding of the optimization mechanism opaque. In this paper, we propose Process Reward Learning (PRL), which decomposes the entropy regularized reinforcement learning objective into intermediate steps, with rigorous process rewards that could be assigned to models accordingly. Starting from theoretical motivation, we derive the formulation of PRL that is essentially equivalent to the objective of reward maximization plus a KL-divergence penalty term between the policy model and a reference model. However, PRL could turn the outcome reward into process supervision signals, which helps better guide the exploration during RL optimization. From our experiment results, we demonstrate that PRL not only improves the average performance for LLMs' reasoning ability measured by average @ n, but also broadens the reasoning boundary by improving the pass @ n metric. Extensive experiments show the effectiveness of PRL could be verified and generalized.
Paper Structure (23 sections, 3 theorems, 60 equations, 2 figures, 6 tables)

This paper contains 23 sections, 3 theorems, 60 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for LLMs
  4. LLMs Reasoning
  5. Process Reward Modeling
  6. Processing Reward Learning from KL Regularized RL
  7. Problem Setting and Formulation
  8. Optimal Process Reward
  9. Process Reward and Policy Learning
  10. Experiments and Results
  11. Experiment Setup
  12. Models and Data
  13. Environment and Computation
  14. Quantitive Results
  15. Ablation Study
  16. ...and 8 more sections

Key Result

theorem 1

Given a reference (normally the original) policy $\pi_0: \Sigma^* \times \Sigma^* \rightarrow [0, 1]$, and the optimial policy maximizing the entropy-regularized reward, $\pi^* = \arg\max_{\pi} \{Q(\pi)\}$, we have:

Figures (2)

  • Figure 1: PRL workflow demonstration. For each prompt and response trajectory $(x,a)$ with $a=[a^1,a^2,\cdots,a^L]$, we could split the reasoning response into several intermediate steps (by fixed length, newline symbol, etc.) and calculate the process reward as the entropy ratio between the current policy $\pi_\omega$ and reference policy $\pi_0$. The final process reward for each step $i$ is the combination of ① the entropy ratio $\sum_{j=1}^p\log\frac{\pi_\omega(a^j|x,a^{(j-1)})}{\pi_0(a^j|x,a^{(j-1)})}$ and ② the final outcome reward $r^*(x,a)$.
  • Figure 2: The training dynamics of KL loss and entropy loss with Qwen2.5-Math-7B as the base model under different configurations.

Theorems & Definitions (3)

  • theorem 1
  • corollary 1
  • theorem 2