SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning

TL;DR

This work tackles the data bottleneck and noise in process reward model (PRM) training by studying Monte Carlo (MC) annotation noise and introducing Self-Denoising Monte Carlo Annotation (SCAN). SCAN combines an efficient data-synthesis workflow that leverages lightweight annotators and a noise-tolerant, confidence-aware learning objective to train PRMs on synthetic MC data. The approach yields strong results on Best-of-N and ProcessBench benchmarks, achieving competitive performance with substantially smaller synthetic datasets and enabling self-improvement through scaling. The findings highlight the potential for scalable, cost-efficient PRM training without heavy reliance on external critic models, with implications for robust reasoning in LLMs across domains, including mathematics and general-domain reasoning like GPQA-Diamond.

Abstract

Process reward models (PRMs) offer fine-grained, step-level evaluations that facilitate deeper reasoning processes in large language models (LLMs), proving effective in complex tasks like mathematical reasoning. However, developing PRMs is challenging due to the high cost and limited scalability of human-annotated data. Synthetic data from Monte Carlo (MC) estimation is a promising alternative but suffers from a high noise ratio, which can cause overfitting and hinder large-scale training. In this work, we conduct a preliminary study on the noise distribution in synthetic data from MC estimation, identifying that annotation models tend to both underestimate and overestimate step correctness due to limitations in their annotation capabilities. Building on these insights, we propose Self-Denoising Monte Carlo Annotation (SCAN), an efficient data synthesis and noise-tolerant learning framework. Our key findings indicate that: (1) Even lightweight models (e.g., 1.5B parameters) can produce high-quality annotations through a self-denoising strategy, enabling PRMs to achieve superior performance with only 6% the inference cost required by vanilla MC estimation. (2) With our robust learning strategy, PRMs can effectively learn from this weak supervision, achieving a 39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench. Despite using only a compact synthetic dataset, our models surpass strong baselines, including those trained on large-scale human-annotated datasets such as PRM800K. Furthermore, performance continues to improve as we scale up the synthetic data, highlighting the potential of SCAN for scalable, cost-efficient, and robust PRM training.

Figures (5)

• Figure 1: Noise distribution analysis of Llama3.1-8B-Instruct and Qwen2.5-Math-7B-Instruct. Left: Overall distribution of noise samples across varying self-confidence levels. Middle: Noise distribution in predicted positive samples where $t_{pred} = \text{inf}$. Right: Distance distribution between $t_{pred}$ and $t_{true}$ for inaccurate negative samples. Additional results of more models can be found in Figure \ref{['fig:noise_distribution_more']}.
• Figure 2: Overview of our data synthesis and robust training framework.
• Figure 3: Ablation results in BoN evaluation and ProcessBench. For ProcessBench, we directly calculate the overall F1 score of full samples. Left: Scaling curve of the PRM training of different datasets. Middle: Scaling curve of selection of tolerance distance. Right: Effectiveness of each component. "Baseline" here represents the vanilla MC estimation method.
• Figure 4: Noise distribution of additional models. Similar observations can be concluded across these models, further validating the consistency of our findings.
• Figure 5: Left: Ablation results of different responses per question estimating self-confidence value. Right: Ablation results on external data sources.