Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs

TL;DR

The paper investigates whether explicit process-level supervision is necessary for robust reasoning in LLMs. It shows that pure RL training on mathematical tasks yields emergent PRM-like capabilities, with RL-trained models surpassing PRM-baselines in process judgments. External PRMs offer limited or negative benefits for these strong models, while a Self-PRM approach—relying on the model’s own internal rewards—consistently improves performance, especially with larger sample sizes, albeit with precision limitations on hard problems. Overall, the work argues that PRM capabilities can emerge from RL, suggesting a scalable path toward more reliable, self-aware complex reasoning models.

Abstract

The development of reasoning capabilities represents a critical frontier in large language models (LLMs) research, where reinforcement learning (RL) and process reward models (PRMs) have emerged as predominant methodological frameworks. Contrary to conventional wisdom, empirical evidence from DeepSeek-R1 demonstrates that pure RL training focused on mathematical problem-solving can progressively enhance reasoning abilities without PRM integration, challenging the perceived necessity of process supervision. In this study, we conduct a systematic investigation of the relationship between RL training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions of reasoning that co-evolve synergistically during pure RL training. In particular, current PRMs underperform simple baselines like majority voting when applied to state-of-the-art models such as DeepSeek-R1 and QwQ-32B. To address this limitation, we propose Self-PRM, an introspective framework in which models autonomously evaluate and rerank their generated solutions through self-reward mechanisms. Although Self-PRM consistently improves the accuracy of the benchmark (particularly with larger sample sizes), analysis exposes persistent challenges: The approach exhibits low precision (<10\%) on difficult problems, frequently misclassifying flawed solutions as valid. These analyses underscore the need for continued RL scaling to improve reward alignment and introspective accuracy. Overall, our findings suggest that PRM may not be essential for enhancing complex reasoning, as pure RL not only improves problem-solving skills but also inherently fosters robust PRM capabilities. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models.

Figures (1)

• Figure 1: Problem Solving Accuracy and F1 Results of the RL-trained model on ProcessBench. We leverage Qwen2.5-7b-base for RL training.