GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning

TL;DR

GroundedPRM introduces a tree-guided, fidelity-aware framework for process reward modeling that combines Monte Carlo Tree Search with external tool verification to produce verifiable, step-level supervision for multi-step reasoning. By fusing per-step correctness signals from an external math engine with trajectory-wide feedback into a hybrid reward, and formatting this into rationale-enhanced sequences, GroundedPRM achieves strong data efficiency (40K auto-labeled samples) and competitive, sometimes superior, performance in reward-guided decoding, including surpassing human-annotated baselines. The approach directly addresses noisy rewards, low factual fidelity, and misalignment with step-level objectives in prior PRMs, enabling scalable and interpretable supervision for complex mathematical reasoning. The work demonstrates robust improvements on ProcessBench and across reward-guided tasks, highlighting the practical impact of structured, verifiable supervision in guiding LLM reasoning.

Abstract

Process Reward Models (PRMs) aim to improve multi-step reasoning in Large Language Models (LLMs) by supervising intermediate steps and identifying errors. However, building effective PRMs remains challenging due to the lack of scalable, high-quality annotations. Existing approaches rely on costly human labeling, LLM-based self-evaluation that is prone to hallucination, or Monte Carlo (MC) estimation, which infers step quality solely from rollout outcomes and often introduces noisy, misaligned supervision due to credit misattribution. These issues result in three core limitations: noisy rewards, low factual fidelity, and misalignment with step-level reasoning objectives. To address these challenges, we introduce GroundedPRM, a tree-guided and fidelity-aware framework for automatic process supervision. To reduce reward noise and enable fine-grained credit assignment, we construct structured reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated supervision, we validate each intermediate step using an external tool, providing execution-grounded correctness signals. To combine both step-level validation and global outcome assessment, we design a hybrid reward aggregation mechanism that fuses tool-based verification with MCTS-derived feedback. Finally, we format the reward signal into a rationale-enhanced, generative structure to promote interpretability and compatibility with instruction-tuned LLMs. GroundedPRM is trained on only 40K automatically labeled samples, amounting to just 10% of the data used by the best-performing PRM trained with auto-labeled supervision. Nevertheless, it achieves up to a 26% relative improvement in average performance on ProcessBench. When used for reward-guided greedy search, GroundedPRM outperforms even PRMs trained with human-labeled supervision, offering a scalable and verifiable path toward high-quality process-level reasoning.

Figures (9)

• Figure 1: Overview of the GroundedPRM framework. GroundedPRM constructs reasoning paths via MCTS, where each node corresponds to an LLM-generated step. During simulation, intermediate steps are verified using an external tool, and final answers are checked against ground truth. Step-level and outcome-level correctness signals are aggregated into a rollout reward, which is backpropagated along the tree to update node statistics; the next node is then selected by UCT, continuing the MCTS search until convergence or budget exhaustion. The framework enables verifiable, interpretable, and structure-aware process supervision for multi-step reasoning. The generative rationale provides interpretable feedback for each step.
• Figure 2: Prompt used to generate the next reasoning step during MCTS rollouts. The output consists of a structured step objective and a logically grounded action aligned with the current goal. These step-level generations are used to construct diverse reasoning trajectories for reward modeling in GroundedPRM.
• Figure 3: Prompt used for tool-based step-level verification. The assistant analyzes the reasoning step for logical consistency, evaluates the relevance of Wolfram Alpha responses, and outputs a binary correctness label along with a structured rationale, forming the fidelity-aware supervision signal for GroundedPRM.
• Figure 4: Comparison between GPT-4o and GroundedPRM as judge models. GPT-4o incorrectly validates the miscalculated reasoning step (570 instead of 630), while GroundedPRM correctly re-evaluates the arithmetic, detects the error, and provides a rationale-supported supervision label, demonstrating fidelity-aware verification.
• Figure 5: Comparison between GPT-4o and GroundedPRM as judge models. After correctly aggregating the three sums to $210$, the LLM performs an unjustified halving to $105$. GPT-4o incorrectly validates this step. GroundedPRM correctly re-evaluates the arithmetic to $210$, verifies each sub-sum, localizes the first wrong step (the halving), and outputs a rationale-supported negative label, showing fidelity-aware, error-type–aware verification.