On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Δ\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Δ\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $Δ\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $Δ\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

Figures (16)

• Figure 1: (a) Token-level metrics for analyzing RLVR updates. (b) Histograms of each metric on responses generated by base and RLVR models. With a log-scale y-axis, most values concentrate near zero for all metrics, but only $\Delta \log p$ shows a directional shift distinguishing RLVR from the base model. (c) Token‑replacement performance: replacing base tokens with RLVR choices at positions selected by each metric, where $\Delta \log p$ recovers RLVR performance with the fewest replacements.
• Figure 2: Token‑replacement performance across metrics and model pairs. While all metrics can recover RLVR‑level accuracy, $\Delta\log p$ does so with the fewest replacements, demonstrating its precision in isolating the reasoning-critical minor tokens changed by RLVR training.
• Figure 3: (a) Token probability and gradient norm coefficient $1-\pi_\theta(\cdot)$ of a DAPO step, where the gradient concentrates on rare, low-probability tokens. (b) Token probability within different $\Delta\log p$ bins, where higher $\Delta\log p$ bins contain lower probability for both base and RLVR models. (c) Effect of top-p filtering on RLVR training performance. Performance declines with more filtering.
• Figure 4: Extrapolation Performance
• Figure 5: Training curves for different reweighting methods on Qwen2.5-Math-7B.

Theorems & Definitions (4)

• Lemma 3.1
• Theorem 4.1
• proof : Proof of Lemma \ref{['lemma:grad']}
• proof : Proof of Theorem \ref{['theorem:extra']}