Towards Robust Process Reward Modeling via Noise-aware Learning

TL;DR

This work tackles noise in process reward modeling caused by Monte Carlo Estimation’s policy-dependent labels. It introduces a two-stage framework: Reflection-Aware Label Correction to suppress false positives caused by self-correction, and Noise-Aware Iterative Training (NAIT) to refine noisy labels using the model’s own confidence across iterations. Empirical results show substantial improvements in step-level discrimination (up to 27 percentage points in average F1) and competitive gains in test-time reasoning, even with limited supervision. The approach demonstrates robust performance across datasets and model scales, highlighting practical gains for scalable, reliable process-level supervision in LLM reasoning. The combination of LLM-based judgment for labeling and iterative, confidence-driven label refinement yields data-efficient, noise-robust PRMs with strong downstream impact.

Abstract

Process Reward Models (PRMs) have achieved strong results in complex reasoning, but are bottlenecked by costly process-level supervision. A widely used alternative, Monte Carlo Estimation (MCE), defines process rewards as the probability that a policy model reaches the correct final answer from a given reasoning step. However, step correctness is an intrinsic property of the reasoning trajectory, and should be invariant to policy choice. Our empirical findings show that MCE producing policy-dependent rewards that induce label noise, including false positives that reward incorrect steps and false negatives that penalize correct ones. To address above challenges, we propose a two-stage framework to mitigate noisy supervision. In the labeling stage, we introduce a reflection-aware label correction mechanism that uses a large language model (LLM) as a judge to detect reflection and self-correction behaviors related to the current reasoning step, thereby suppressing overestimated rewards. In the training stage, we further propose a \underline{\textbf{N}}oise-\underline{\textbf{A}}ware \underline{\textbf{I}}terative \underline{\textbf{T}}raining framework that enables the PRM to progressively refine noisy labels based on its own confidence. Extensive Experiments show that our method substantially improves step-level correctness discrimination, achieving up to a 27\% absolute gain in average F1 over PRMs trained with noisy supervision.

Figures (9)

• Figure 1: MCE method shows significant deviation compared to human-annotated ground truth, regardless of model capability and sampling budget.
• Figure 2: Illustration of MCE label noise. (Left) False Positive: A high reward is assigned to an incorrect step due to the policy's subsequent self-correction. (Right) False Negative: A low reward is assigned to a correct step due to a subsequent policy failure.
• Figure 3: Overview of the proposed framework for robust process reward modeling. The framework consists of two stages. (Top right) In the data annotation stage, we propose a reflection-aware label correction mechanism, where an LLM judge detects reflection and self-correction behaviors within reasoning trajectories to identify and suppress overestimated rewards. (Bottom) In the training stage, we introduce a noise-aware iterative training framework that trains the process reward model with noise-robust objectives and progressively refines noisy labels based on model confidence across multiple stages.
• Figure 4: Effect of training data scale and iterative training on PRM performance. (a) The impact of data scale on PRM's F1 score on ProcessBench; (b) The impact of data scale on PRM's accuracy at PRM@8; (c) The impact of the number of training epochs on PRM's F1 score at ProcessBench; (d) The impact of the number of training epochs on PRM's accuracy at PRM@8.
• Figure 5: The MC-based PRM mislabels a correct step, whereas MCRD-based and NAIT-based PRMs assign high scores.