Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning

Massimiliano Pronesti, Anya Belz, Yufang Hou

TL;DR

This work introduces Verifiable Process Reward Models (VPRMs), a reinforcement-learning paradigm where intermediate reasoning steps are checked by deterministic, rule-based verifiers. By applying VPRMs to risk-of-bias assessment in medical systematic reviews, the authors provide theoretical guarantees that correct reasoning trajectories receive positive gradient signals and incorrect ones receive negative signals, under mild conditions. Empirically, VPRMs outperform both outcome-only RLVR and neural PRMs across multiple RoB benchmarks, achieving higher accuracy, macro-F1, and coherence, with substantial gains in evidence grounding and logical consistency. The approach offers a practical, transparent alternative to neural judges for structured reasoning tasks, advancing reliable, verifiable AI-assisted evidence evaluation in healthcare and beyond.

Abstract

Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.

Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning

TL;DR

This work introduces Verifiable Process Reward Models (VPRMs), a reinforcement-learning paradigm where intermediate reasoning steps are checked by deterministic, rule-based verifiers. By applying VPRMs to risk-of-bias assessment in medical systematic reviews, the authors provide theoretical guarantees that correct reasoning trajectories receive positive gradient signals and incorrect ones receive negative signals, under mild conditions. Empirically, VPRMs outperform both outcome-only RLVR and neural PRMs across multiple RoB benchmarks, achieving higher accuracy, macro-F1, and coherence, with substantial gains in evidence grounding and logical consistency. The approach offers a practical, transparent alternative to neural judges for structured reasoning tasks, advancing reliable, verifiable AI-assisted evidence evaluation in healthcare and beyond.

Abstract

Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.
Paper Structure (51 sections, 25 equations, 7 figures, 8 tables, 1 algorithm)

This paper contains 51 sections, 25 equations, 7 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Policy Optimisation Algorithms
  4. Group Relative Policy Optimization (GRPO)
  5. Dynamic sAmpling Policy Optimization (DAPO)
  6. Rule-based Reward Modeling
  7. Systematic Reviews
  8. Risk of Bias Assessment
  9. Verifiable Process Reward Models
  10. Reasoning Trajectories and Steps
  11. Verifiers and Process Rewards
  12. Reward Separation and Optimisation Guarantee
  13. Experiments
  14. Training Dataset Creation
  15. Synthetic Data Annotation for Structural Reasoning Processes
  16. ...and 36 more sections

Figures (7)

  • Figure 1: Illustration of the verifiable reasoning setup for risk-of-bias assessment (type A: bias arising from the randomisation process). Top: given an input study $x$, the model produces a structured reasoning trace $Y=(o_1,\ldots,o_T)$ with step-level labels $(\ell_1,\ell_2,\ell_3,\ell_4)$, each corresponding to a guideline-defined assessment question. Bottom: the corresponding rule-based decision tree, which deterministically maps each combination of step-level labels to low (+), high (-), or moderate (?) risk.
  • Figure 2: Comparison between verifiable outcome rewards (left), which evaluates only the final risk label, and verifiable process rewards (right), which additionally verifies each reasoning step and its associated label.
  • Figure 3: Reward dynamics. Format rewards plateau early, while accuracy and process rewards improve gradually, indicating that LLMs quickly learn structure but continue to refine quality.
  • Figure 4: Prompt for synthetic data annotation.
  • Figure 5: Steps and labels.
  • ...and 2 more figures