Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proof-RM: A Scalable and Generalizable Reward Model for Math Proof

Haotong Yang, Zitong Wang, Shijia Kang, Siqi Yang, Wenkai Yu, Xu Niu, Yike Sun, Yi Hu, Zhouchen Lin, Muhan Zhang

TL;DR

ProofRM tackles the challenge of verifying full mathematical proofs by moving beyond verifiable final answers to a scalable, learnable reward signal for proof correctness. It builds a diverse QPC data pipeline that combines real problems with LLM-generated proofs and composition-level human checks, enabling robust RLVR-based training of a proof verifier without relying solely on expert annotation. A stability-focused RL recipe—with LLM-for-RM supervision and balanced token weighting—yields a proof-check RM that generalizes to new problem distributions and supports test-time scaling via best-of-$k$ selection. Empirical results show ProofRM outperforms strong baselines and frontier LLMs on in-distribution and out-of-distribution probes, offering practical recipes for enhancing LLM mathematical reasoning and proof verification at scale.

Abstract

While Large Language Models (LLMs) have demonstrated strong math reasoning abilities through Reinforcement Learning with *Verifiable Rewards* (RLVR), many advanced mathematical problems are proof-based, with no guaranteed way to determine the authenticity of a proof by simple answer matching. To enable automatic verification, a Reward Model (RM) capable of reliably evaluating full proof processes is required. In this work, we design a *scalable* data-construction pipeline that, with minimal human effort, leverages LLMs to generate a large quantity of high-quality "**question-proof-check**" triplet data. By systematically varying problem sources, generation methods, and model configurations, we create diverse problem-proof pairs spanning multiple difficulty levels, linguistic styles, and error types, subsequently filtered through hierarchical human review for label alignment. Utilizing these data, we train a proof-checking RM, incorporating additional process reward and token weight balance to stabilize the RL process. Our experiments validate the model's scalability and strong performance from multiple perspectives, including reward accuracy, generalization ability and test-time guidance, providing important practical recipes and tools for strengthening LLM mathematical capabilities.

Proof-RM: A Scalable and Generalizable Reward Model for Math Proof

TL;DR

ProofRM tackles the challenge of verifying full mathematical proofs by moving beyond verifiable final answers to a scalable, learnable reward signal for proof correctness. It builds a diverse QPC data pipeline that combines real problems with LLM-generated proofs and composition-level human checks, enabling robust RLVR-based training of a proof verifier without relying solely on expert annotation. A stability-focused RL recipe—with LLM-for-RM supervision and balanced token weighting—yields a proof-check RM that generalizes to new problem distributions and supports test-time scaling via best-of- selection. Empirical results show ProofRM outperforms strong baselines and frontier LLMs on in-distribution and out-of-distribution probes, offering practical recipes for enhancing LLM mathematical reasoning and proof verification at scale.

Abstract

While Large Language Models (LLMs) have demonstrated strong math reasoning abilities through Reinforcement Learning with *Verifiable Rewards* (RLVR), many advanced mathematical problems are proof-based, with no guaranteed way to determine the authenticity of a proof by simple answer matching. To enable automatic verification, a Reward Model (RM) capable of reliably evaluating full proof processes is required. In this work, we design a *scalable* data-construction pipeline that, with minimal human effort, leverages LLMs to generate a large quantity of high-quality "**question-proof-check**" triplet data. By systematically varying problem sources, generation methods, and model configurations, we create diverse problem-proof pairs spanning multiple difficulty levels, linguistic styles, and error types, subsequently filtered through hierarchical human review for label alignment. Utilizing these data, we train a proof-checking RM, incorporating additional process reward and token weight balance to stabilize the RL process. Our experiments validate the model's scalability and strong performance from multiple perspectives, including reward accuracy, generalization ability and test-time guidance, providing important practical recipes and tools for strengthening LLM mathematical capabilities.
Paper Structure (43 sections, 24 equations, 6 figures, 12 tables)

This paper contains 43 sections, 24 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Motivation: asymmetry of verification
  3. Previous success based on strong asymmetry
  4. Limits of current asymmetry on math
  5. Some attempt to verify math proof
  6. Checking the proof has asymmetry of verification
  7. Scalable and Generalized RM Training
  8. Automated Diverse Data Collection
  9. Question-proof source diversity
  10. Proof generation method diversity
  11. LLM diversity
  12. LLM-aided consistent labeling with composition-level human check
  13. Stable RLVR algorithm
  14. Experiments
  15. Reward accuracy
  16. ...and 28 more sections

Figures (6)

  • Figure 1: Pipeline of data collection and training of ProofRM.
  • Figure 2: Training data distribution generated by the data pipeline.
  • Figure 3: The training curve shows an increasing trend of the average reward, matching the trend of the testing accuracy.
  • Figure 4: Best-of-k curves for different models. Here the Ours model is ProofRM-8B.
  • Figure 5: Output length comparison of different token weighting strategies
  • ...and 1 more figures