Not All Steps are Informative: On the Linearity of LLMs' RLVR Training

TL;DR

The paper identifies a strong, universal linearity in RLVR training, with both weights and token log-probabilities evolving nearly linearly with training steps across models and algorithms. Building on this, it introduces Logits Extrapolation, Weight Extrapolation, and RL-Extra to forecast future model states or to interleave extrapolation with actual RL updates, achieving up to 3% gains and as much as 6.1x wall-clock speedups on standard math and coding benchmarks. The methods maintain performance comparable to or better than standard RL while reducing compute, though linearity breaks down for very long horizons, motivating a hybrid approach. These findings offer practical acceleration for LLM post-training and deepen understanding of RLVR dynamics, while acknowledging limitations in scale, multi-turn RL, and deployment in industry.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a central component of large language model (LLM) post-training. Unlike supervised fine-tuning (SFT), RLVR lets an LLM generate multiple candidate solutions and reinforces those that lead to a verifiably correct final answer. However, in practice, RLVR often requires thousands of training steps to reach strong performance, incurring substantial computation largely attributed to prolonged exploration. In this work, we make a surprising observation: during RLVR, LLMs evolve in a strongly linear manner. Specifically, both model weights and model output log-probabilities exhibit strong linear correlations with RL training steps. This suggests that RLVR predominantly amplifies trends that emerge early in training, rather than continuously discovering new behaviors throughout the entire optimization trajectory. Motivated by this linearity, we investigate whether future model states can be predicted from intermediate checkpoints via extrapolation, avoiding continued expensive training. We show that Weight Extrapolation produces models with performance comparable to standard RL training while requiring significantly less computation. Moreover, Logits Extrapolation consistently outperforms continued RL training on all four benchmarks by extrapolating beyond the step range where RL training remains stable.

Figures (9)

• Figure 1: Linearity analysis for model weights and outputs during RLVR training. (a) and (b) show the distributions of $R^2$ for weight and token log-probabilities, respectively. Both distributions are concentrated around 0.9, indicating strong linearity. (c) plots the trajectories of four randomly selected weights, and (d) shows token log-probability changes at four example positions. The log-probabilities of “wait” and “but” increase over RL steps, suggesting more reflection and revision, whereas those of “earlier” and “alternatively” decrease, indicating reduced need for backtracking and branching.
• Figure 2: Linearity consistency across diverse experimental setups. The $R^2$ scores consistently exceed 0.7 (dashed line) across various base models (e.g., DS-Qwen, DS-Llama), scale sizes (1.5B to 8B), and training algorithms (GSPO, Reinforce++, and GRPO). The high $R^2$ values for both token log-probabilities and weights indicate a robust linear relationship that persists across architectural and algorithmic configurations.
• Figure 3: The source of output changes in a representative LLM layer.
• Figure 4: Accuracy comparison on AIME and LCB benchmarks. Logit Extrapolation yields consistent improvements over standard RL across all evaluated settings.
• Figure 5: Weight Extrapolation performance on AIME24 across different target steps.