GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

TL;DR

<3-5 sentence high-level summary>GenPRM reframes Process Reward Models as generative systems that perform explicit Chain-of-Thought reasoning with integrated code verification, addressing limited supervision and non-generative training signals in traditional PRMs while enabling test-time scaling. It introduces Relative Progress Estimation to obtain reliable process-supervision labels and a rationale-synthesis pipeline that yields high-quality data for training. Empirical results on ProcessBench and mathematical reasoning tasks show GenPRM achieves state-of-the-art performance with relatively small training data (23K from MATH) and can scale at test time to surpass larger models such as GPT-4o and 72B PRMs. Additionally, GenPRM functions as both a verifier and a critic, offering a new paradigm that connects PRMs with critic-based refinement in LLMs and enabling broader adoption of generative process supervision.

Abstract

Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs. Our code, model, and data will be available in https://ryanliu112.github.io/GenPRM.

Figures (6)

• Figure 1: GenPRM achieves state-of-the-art performance across multiple benchmarks in two key roles: (a) As a verifier: GenPRM-7B outperforms all classification-based PRMs of comparable size and even surpasses Qwen2.5-Math-PRM-72B via test-time scaling. (b) As a critic: GenPRM-7B demonstrates superior critique capabilities, achieving 3.4× greater performance gains than DeepSeek-R1-Distill-Qwen-7B after 3 refinement iterations.
• Figure 2: Comparison between GenPRM (right) and previous classification-based PRMs (left).
• Figure 3: Overall framework of GenPRM. Our framework consists of six key parts: ① The policy model generates solution steps, with MC scores estimated from rollout trajectories. ② Our proposed RPE derives accurate PRM labels. ③ High-quality process supervision data is synthesized through CoT reasoning augmented with code verification. ④ We apply consensus filtering followed by SFT to train GenPRM. ⑤ The trained GenPRM functions as a verifier or critic, enabling enhanced test-time scaling for policy models. ⑥ The performance of GenPRM further improves through test-time scaling.
• Figure 4: BoN results with different generation models on multiple mathematical benchmarks.
• Figure 5: The first case of GenPRM.