When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards

TL;DR

The paper investigates whether reinforcement learning with verifiable rewards (RLVR) yields genuinely new capabilities or merely sharpens existing knowledge. It identifies two mechanisms—finite‑batch sampling bias and semantic coupling—that cause over‑sharpening and collapse of the solution space, and it formalizes this through a probabilistic/NTK‑styled analysis. To mitigate collapse, the authors propose inverse‑success advantage calibration (IAC) and distribution‑level calibration (DLC) using a memory network to diversify sampling; they also analyze an empirical toy model to illustrate the dynamics. Across six mathematical benchmarks and multiple Qwen backbones, the calibrated approaches improve AVG@8 and PASS@8, with IAC often delivering strong gains and DLC providing additional benefits when compute permits, all while preserving higher policy entropy relative to baselines. These results suggest that careful calibration can maintain diverse, robust reasoning patterns in RLVR, enhancing generalization in logic‑heavy domains.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a central paradigm for turning large language models (LLMs) into reliable problem solvers, especially in logic-heavy domains. Despite its empirical success, it remains unclear whether RLVR elicits novel capabilities or merely sharpens the distribution over existing knowledge. We study this by formalizing over-sharpening, a phenomenon where the policy collapses onto limited modes, suppressing valid alternatives. At a high level, we discover finite-batch updates intrinsically bias learning toward sampled modes, triggering a collapse that propagates globally via semantic coupling. To mitigate this, we propose inverse-success advantage calibration to prioritize difficult queries and distribution-level calibration to diversify sampling via a memory network. Empirical evaluations validate that our strategies can effectively improve generalization.

Figures (6)

• Figure 1: Effect of advantage estimation and group size on over-sharpening. Probability of Cat and Persian over training for raw (left), mean-shifted (mid), and normalized (right) advantages.
• Figure 2: Semantic coupling accelerates collapse. Accuracy on the held-out Siamese class while training only on Persian. Stronger semantic coupling and larger group sizes both lead to faster degradation of Siamese accuracy.
• Figure 3: The effectiveness of RLOO and Reinforce++ with baseline on over-sharpening.
• Figure 4: Effect of different optimizers on over-sharpening.
• Figure 5: We apply our proposed mitigations to the Softmax example. The left column (a, c) employs only inverse-Success advantage calibration, showing a delayed rate of collapse. The right column (b, d) incorporates distribution-level calibration (memory network, $\mu=0.5$), which arrests collapse and maintains diversity, even under the aggressive updates of AdamW.

Theorems & Definitions (17)

• Definition 2.1
• Theorem 2.1
• Corollary 2.1
• proof
• Theorem 2.2
• Theorem 2.3
• Theorem 2.4
• Corollary 2.2
• Theorem 1.1
• proof