PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards

TL;DR

PRISM introduces a unified post-training framework for large language models that operates without ground-truth rewards by combining a Process Reward Model (PRM) with the model's own internal confidence signals. It demonstrates that internal signals alone can be unstable and prone to reward hacking, while PRMs (via GenPRM) provide reliable process-level feedback. By fusing PRM rewards with self-certainty through a normalized, gamma-weighted combination, PRISM achieves stable training and improved performance on math and code reasoning benchmarks, approaching or surpassing ground-truth-reward baselines in several settings. The work highlights a practical path toward scalable, label-free post-training of LLMs, while noting limitations related to PRM quality, latency, and generalization beyond the tested domains.

Abstract

Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check.

Figures (10)

• Figure 1: Left: An overview of our proposed PRISM framework. When ground-truth rewards are absent, LLMs learn from their intrinsic signal and utilize feedback on their reasoning process to keep its intrinsic internal signal in check. Right: pass@1 accuracy comparison (see Tab. \ref{['tab:acc_math']} for comprehensive comparison) of Qwen2.5-3B and Qwen2.5-7B: Base, GRPO (on ground truth), INTUITOR, and PRISM (ours) on three different math benchmarks. PRISM outperforms INTUITOR, and closely matches GRPO across all benchmarks while using no ground-truth rewards; GRPO is included as a strong reference when such rewards are available.
• Figure 2: Mean accuracy (left) and mean length (right) of the training rollouts under three different RLIF methods. Initial trend shows rapid increase in mean accuracy across all methods, but the models start to degrade as the training progresses. Mean length of the generations start to increase and correspond to the step when the accuracy starts to drop. The base model is Qwen2.5-3B and the training data is MATH.
• Figure 3: Top: Rolling correlation between true accuracy and the respective proxy rewards during training across all methods. All training were run for total of 300 optimization steps ($\approx$ 6 epochs) on MATH dataset. Dashed lines denote mean correlation. Bottom: Mean accuracy vs mean self-certainty of training rollouts. Self-certainty score does not collapse with the true accuracy.
• Figure 4: Distribution of self-certainty scores of responses generated by (a) Qwen2.5-3B base model and (b) Qwen2.5-3B INTUITOR trained model. (c) Distribution of PRM rewards for responses generated by Qwen2.5-3B base model. $U$ is the Mann-Whitney U-test score which quantifies the separation between two distributions. $p$ and $r$ are the $p$-value and effect-size respectively. Unlike self-certainty, PRM rewards reliably predict correct from incorrect responses. Dashed line represent mean of the respective distribution.
• Figure 5: (left) Mean accuracy and PRM rewards of the training rollouts and (right) mean length of rollouts.