Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering

Wenlin Zhong, Chengyuan Liu, Yiquan Wu, Bovin Tan, Changlong Sun, Yi Wang, Xiaozhong Liu, Kun Kuang

TL;DR

P2S tackles the scarcity of verifiable rewards in general-domain reasoning by introducing Probabilistic Process Supervision, a self-supervised framework that provides dense, step-level supervision through dynamic Gold-CoT synthesis and the Path Faithfulness Reward (PFR). PFR measures the information gain of reasoning prefixes relative to a reference Gold-CoT, and is weighted and integrated with outcome-based rewards in a hierarchical scheme to combat reward sparsity. The approach eliminates the need for external reward models or human-annotated reasoning steps, and experiments on DROP and Medical QA show that P2S outperforms strong baselines, including verifier-based and RLPR methods, across both lexical and semantic metrics. The work demonstrates that dense process supervision can significantly improve general-domain reasoning, offering a verifier-free, scalable path toward more reliable open-ended QA systems.

Abstract

While reinforcement learning with verifiable rewards (RLVR) has advanced LLM reasoning in structured domains like mathematics and programming, its application to general-domain reasoning tasks remains challenging due to the absence of verifiable reward signals. To this end, methods like Reinforcement Learning with Reference Probability Reward (RLPR) have emerged, leveraging the probability of generating the final answer as a reward signal. However, these outcome-focused approaches neglect crucial step-by-step supervision of the reasoning process itself. To address this gap, we introduce Probabilistic Process Supervision (P2S), a novel self-supervision framework that provides fine-grained process rewards without requiring a separate reward model or human-annotated reasoning steps. During reinforcement learning, P2S synthesizes and filters a high-quality reference reasoning chain (gold-CoT). The core of our method is to calculate a Path Faithfulness Reward (PFR) for each reasoning step, which is derived from the conditional probability of generating the gold-CoT's suffix, given the model's current reasoning prefix. Crucially, this PFR can be flexibly integrated with any outcome-based reward, directly tackling the reward sparsity problem by providing dense guidance. Extensive experiments on reading comprehension and medical Question Answering benchmarks show that P2S significantly outperforms strong baselines.

P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering

TL;DR

P2S tackles the scarcity of verifiable rewards in general-domain reasoning by introducing Probabilistic Process Supervision, a self-supervised framework that provides dense, step-level supervision through dynamic Gold-CoT synthesis and the Path Faithfulness Reward (PFR). PFR measures the information gain of reasoning prefixes relative to a reference Gold-CoT, and is weighted and integrated with outcome-based rewards in a hierarchical scheme to combat reward sparsity. The approach eliminates the need for external reward models or human-annotated reasoning steps, and experiments on DROP and Medical QA show that P2S outperforms strong baselines, including verifier-based and RLPR methods, across both lexical and semantic metrics. The work demonstrates that dense process supervision can significantly improve general-domain reasoning, offering a verifier-free, scalable path toward more reliable open-ended QA systems.

Abstract

While reinforcement learning with verifiable rewards (RLVR) has advanced LLM reasoning in structured domains like mathematics and programming, its application to general-domain reasoning tasks remains challenging due to the absence of verifiable reward signals. To this end, methods like Reinforcement Learning with Reference Probability Reward (RLPR) have emerged, leveraging the probability of generating the final answer as a reward signal. However, these outcome-focused approaches neglect crucial step-by-step supervision of the reasoning process itself. To address this gap, we introduce Probabilistic Process Supervision (P2S), a novel self-supervision framework that provides fine-grained process rewards without requiring a separate reward model or human-annotated reasoning steps. During reinforcement learning, P2S synthesizes and filters a high-quality reference reasoning chain (gold-CoT). The core of our method is to calculate a Path Faithfulness Reward (PFR) for each reasoning step, which is derived from the conditional probability of generating the gold-CoT's suffix, given the model's current reasoning prefix. Crucially, this PFR can be flexibly integrated with any outcome-based reward, directly tackling the reward sparsity problem by providing dense guidance. Extensive experiments on reading comprehension and medical Question Answering benchmarks show that P2S significantly outperforms strong baselines.
Paper Structure (35 sections, 10 equations, 8 figures, 3 tables)

This paper contains 35 sections, 10 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Reasoning
  4. Reasoning in General Domains
  5. Process Reward Supervision
  6. Preliminaries
  7. Reasoning Optimization With RL
  8. Reinforcement Learning with Reference Probability Reward (RLPR)
  9. Methodology
  10. Problem Definition
  11. Overall Architecture
  12. Dynamic Gold-CoT Synthesis and Filtering
  13. Candidate Synthesis.
  14. Quality-Based Filtering.
  15. Path Faithfulness Reward (PFR)
  16. ...and 20 more sections

Figures (8)

  • Figure 1: Comparing reward mechanisms: P2S rewards the entire reasoning process.
  • Figure 2: An overview of our Probabilistic Process Supervision (P2S) framework. (1) Gold-CoT Synthesis (Top): A dynamic reference path (Gold-CoT) is created by generating and filtering the policy model's own reasoning outputs. (2) PFR Calculation (Bottom): For each new trace, a step-wise Path Faithfulness Reward (PFR) is computed by aligning it against the Gold-CoT. (3) Reward Shaping $\&$ Aggregation: The step-wise rewards are shaped using a sigmoid function to assign progressively higher weights to later reasoning steps. These weighted scores are then summed to produce the final, sample-level Path Faithfulness Reward (PFR) used for policy optimization.
  • Figure 3: P2S outperforms in verifiable tasks
  • Figure 4: Case Study
  • Figure 5: We adopt the training and inference prompt of R1 guo2025deepseek
  • ...and 3 more figures