PACR: Progressively Ascending Confidence Reward for LLM Reasoning

TL;DR

This work tackles the sparsity of terminal rewards in Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning by introducing PACR, a dense, model-intrinsic signal derived from the evolution of the model’s belief in the ground-truth answer. PACR comes in two forms: Sparse-PACR, which rewards trajectory-wide consistency in ground-truth confidence growth, and Dense-PACR, which rewards step-level confidence changes via a normalized, discounted return. The authors provide empirical evidence and a theoretical justification that ground-truth confidence growth is a useful inductive bias, enhancing exploration toward logically sound reasoning paths and enabling faster convergence. Integrating PACR with Group Relative Policy Optimization (GRPO) yields systematic improvements across multiple math benchmarks and model sizes, with Dense-PACR delivering the strongest gains. Overall, PACR demonstrates that dense, internally-sourced feedback can make RLVR training more effective, reliable, and data-efficient for complex reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.

Figures (14)

• Figure 1: Stepwise confidence growth. For a question $q$, a well-formed sequence of reasoning steps $h_1,\ldots,h_k$ should increase the model's probability of the ground-truth answer $Y_{\mathrm{gt}}$ across steps.
• Figure 2: Consistency of confidence growth correlates with correctness. Outcome-correct trajectories ($\mathcal{T}_{\text{correct}}$) exhibit a higher proportion of steps with positive confidence gain ($C_k > 0$) compared to incorrect trajectories ($\mathcal{T}_{\text{incorrect}}$).
• Figure 3: Consistency of confidence growth reflects reasoning quality. Coherent reasoning paths ($\mathcal{T}_{\text{coherent}}$) show a more consistent confidence ascent (higher proportion of $C_k > 0$ steps) than spurious paths ($\mathcal{T}_{\text{spurious}}$), even when both yield the correct final answer.
• Figure 4: Qualitative example of a pivotal step. Among the reasoning steps, a critical insight at step $h_7$ (the introduction of Vieta's formulas for a cubic equation) results in a large, distinct spike in the ground-truth confidence gain ($C_7 = +4.83$). This is substantially larger than the gains from more routine algebraic steps. Further qualitative examples are provided in Appendix \ref{['appendix:obs3_qual']}.
• Figure 5: Quantitative validation of step importance. In a pairwise comparison, an LLM evaluator judged the step with the higher confidence gain ($C_i > C_j$) as more critical with a win rate significantly above chance, confirming that gain magnitude correlates with step importance.

Theorems & Definitions (2)

• Proposition 1
• proof