Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision

TL;DR

This work tackles hallucinations in multi‑step mathematical reasoning by introducing PathFinder‑PRM, a hierarchical, error‑aware process reward model that first types errors at each step (math vs consistency) and then estimates a step reward conditioned on those error signals. It builds a ~400K trajectory dataset with three‑dimensional step labels by combining PRM800K and RLHFlow data, and trains PathFinder‑PRM from a Qwen2.5‑Math‑7B backbone using a two‑pass masked‑token objective that decouples error detection from reward estimation. Empirically, PathFinder‑PRM achieves state‑of‑the‑art PRMScore on PRMBench (67.7) and strong gains on ProcessBench (avg F1 up to 69.5), while delivering improved reward‑guided search (prm@8 = 48.3) and data‑efficient performance relative to larger automated annotation baselines. The results confirm that explicit, fine‑grained error typing and hierarchical supervision yield more reliable process supervision, enabling robust end‑to‑end mathematical reasoning with greater data efficiency.

Abstract

Large Language Models (LLMs) are prone to hallucination, especially during multi-hop and reasoning-intensive tasks such as mathematical problem solving. While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions. We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine-grained signals to estimate step correctness. To train PathFinder-PRM, we construct a 400K-sample dataset by enriching the human-annotated PRM800K corpus and RLHFlow Mistral traces with three-dimensional step-level labels. On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data. When applied to reward guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline. These results demonstrate that decoupled error detection and reward estimation not only boost fine-grained error detection but also substantially improve end-to-end, reward-guided mathematical reasoning with greater data efficiency.

Figures (3)

• Figure 1: Comparing the Methodology of Existing PRMs against PathFinder-PRM.
• Figure 2: Performance comparison of language models across three benchmarks. The PathFinder-PRM-7B model (gray) shows the highest performance across all benchmarks.
• Figure 3: PRMBench Overall PRMscore against the data scales of different baselines and PathFinder-PRM variants.