SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning

TL;DR

SPARK presents a three-stage, reference-free RL pipeline that uses inference-time scaling to generate dense, step-level verifications for LLM solutions, training generative process reward models without ground-truth data. These models then serve as dense rewards in RL, achieving competitive or superior performance to ground-truth-based methods on mathematical reasoning benchmarks. Key contributions include a multi-scale generator-verifier data generation framework, PRM-CoT with step-wise verification, and a thorough analysis of reward-hacking patterns with mitigation strategies. The approach broadens RL applicability to domains lacking verifiable answers by removing reliance on ground-truth references while maintaining strong reasoning performance.

Abstract

Process reward models (PRMs) that provide dense, step-level feedback have shown promise for reinforcement learning, yet their adoption remains limited by the need for expensive step-level annotations or ground truth references. We propose SPARK: a three-stage framework where in the first stage a generator model produces diverse solutions and a verifier model evaluates them using parallel scaling (self-consistency) and sequential scaling (meta-critique). In the second stage, we use these verification outputs as synthetic training data to fine-tune generative process reward models, which subsequently serve as reward signals during training. We show that aggregating multiple independent verifications at the step level produces training data for process reward models that surpass ground-truth outcome supervision, achieving 67.5 F1 on ProcessBench (a benchmark for identifying erroneous steps in mathematical reasoning) compared to 66.4 for reference-guided training and 61.9 for GPT-4o. In the final stage, we apply our generative PRM with chain-of-thought verification (PRM-CoT) as the reward model in RL experiments on mathematical reasoning, and introduce format constraints to prevent reward hacking. Using Qwen2.5-Math-7B, we achieve 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). Our work enables reference-free RL training that exceeds ground-truth methods, opening new possibilities for domains lacking verifiable answers or accessible ground truth.

Figures (10)

• Figure 1: Spark: A three-stage pipeline for reference-free RL training with generative process reward models. Stage I: Generate synthetic verification data using inference-time scaling methods (self-consistency and meta-critique) without ground truth through a multi-scale generator-verifier framework. Stage II: Train three generative reward model variants (ORM, PRM, PRM-CoT) via supervised fine-tuning on the synthetic data. Stage III: Apply trained PRMs in RL with GRPO using different reward designs.
• Figure 2: Multi-scale generator-verifier framework for synthetic verification data generation. The generator produces multiple solutions per problem, and the verifier evaluates them without ground truth using different inference-time scaling methods. Parallel scaling (Self-Consistency): Generates multiple independent verifications and aggregates them through either outcome-level majority voting (voting on final Yes/No verdicts) or step-level majority voting (voting on each step's correctness). Sequential scaling (Meta-Critique): Generates an initial verification, critiques it to identify errors, and merges both into a refined verification.
• Figure 3: Average F1 scores on ProcessBench for PRM variants trained using synthetic data from different inference-time scaling methods. Leftmost bars show Single Verification baseline (no scaling). All PRMs are fine-tuned from Qwen2.5-14B-Instruct.
• Figure 4: Comparison of reference-free PRM-CoT, ground-truth RLVR, and random rewards. Left: Training rewards for (1) PRM-CoT with process-aware rewards (Section \ref{['subsec:rl_method']}), (2) RLVR with ground-truth answer verification, and (3) random rewards via coin flip (50% probability) independent of correctness. Right: Average test accuracy on MATH-500, AIME 2024, and AIME 2025. PRM-CoT consistently outperforms RLVR while spurious random rewards fail to improve from baseline.
• Figure 5: Comparison of generative reward models during RL training. Average test accuracy (MATH-500, AIME 2024, AIME 2025) for three variants trained with step-level consistency and used with process-aware rewards (Section \ref{['subsec:rl_method']}). PRM-CoT with chain-of-thought verification consistently outperforms direct step judgment (PRM) and outcome-only verification (ORM).