Adversarial Training for Process Reward Models

TL;DR

This work introduces Adversarially Trained Process Reward Models (APRM), a two-player, general-sum framework where a Generator learns to craft plausible yet incorrect reasoning steps to challenge a Process Reward Model that simultaneously learns to detect these errors. The authors establish theoretical guarantees, including Nash Equilibrium existence and linear convergence under regularized utilities and Optimistic Gradient Descent-Ascent, and implement a PPO-based training loop augmented with OGDA. Empirically, APRM yields consistent improvements in solver accuracy across math benchmarks, scales with solver capacity, generalizes to cross-domain reasoning, and provides a stronger reward signal for RL post-training compared to static or outcome-based baselines. Ablation and qualitative analyses reveal the progression of harder negatives and the robustness of APRM against stronger solvers, while also identifying failure modes such as preconditioned theorem checks. Overall, APRM demonstrates a dynamic, adaptable curriculum for PRMs that fosters improved reasoning accuracy and generalization with manageable theoretical and empirical overheads.

Abstract

Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (\texttt{APRM}), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, \texttt{APRM} improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. \texttt{APRM} achieves gains of $+5.3$ pp on out-of-distribution tasks.

Figures (4)

• Figure 1: APRM Overview. The Generator $G_\theta$ perturbs the correct step $s$ into an incorrect step $s'$. The PRM $R_\phi$ evaluates a step for it's correctness given the question $Q$ and partial solution $S$. Both $G_\theta$ and $R_\phi$ optimize for rewards given based on the perturbed step and $R_\phi$'s prediction, details in section \ref{['sec:reward_definitions']}.
• Figure 2: APRM augmented solver accuracy averaged across math reasoning benchmarks for different $R_\phi$ and $G_\theta$ backbones.
• Figure 3: Training compute vs. accuracy on Math and SciBench. Dotted arrows connect ReST-MCTS (60 h) to APRM (72 h), showing the direction of improvement. APRM achieves higher accuracy on both benchmarks at slightly higher compute.
• Figure 4: Comparison of performance on a subset of PRM800K prm800k. APRM (hatched) achieves the highest accuracy, precision, recall, and F1 score, indicating stronger alignment with humans

Theorems & Definitions (12)

• Lemma 1: Equivalence of KL Penalty and Entropy Bonus
• Lemma 2: Form of the Regularized Optimum
• proof
• Definition 1: Support of a Probability Distribution
• Definition 2: Full Support of an Equilibrium Policy
• Theorem 1: Full Support of the Equilibrium Policy
• proof
• Corollary 1: Avoidance of Mode Collapse
• Definition 3: Strong Monotonicity
• Definition 4: Lipschitz Continuity