Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

TL;DR

Reinforcement learning with verifiable rewards (RLVR) can improve performance on contaminated benchmarks even with spurious rewards, suggesting memorization shortcuts rather than genuine reasoning. The authors apply a mechanistic suite (Path Patching, Logit Lens, Jensen-Shannon Divergence, Neural Ordinary Differential Equations) to Qwen 2.5-Math to locate and validate a memory circuit. They identify a Functional Anchor in layers 18–20 and Structural Adapters in layers 21+, demonstrate a Perplexity Paradox, and prove causal control by scaling MLP keys to amplify or suppress contamination-driven outputs. This work provides a concrete pathway to detect, understand, and mitigate data contamination in RLVR-tuned models, with practical implications for safer evaluation and deployment of LLMs in reasoning domains.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.

Figures (25)

• Figure 1: Left: Overall accuracy of four models on six benchmarks. Right: Dataset-selection rationale. Based on the accuracy gap, we retain MATH500, MinervaMath and LiveMathBench as our principal evaluation suites. Questions that are wrong before RLVR but correct after are treated as leaked and are the focus of subsequent mechanistic tests.
• Figure 2: Partial Prompt Evaluation for Qwen2.5-Math-7B. ROUGE-L scores (a) and completion accuracy (b) before (dashed) and after (solid) spurious RLVR. We analyze the "Wrong$\rightarrow$Right" group (green), representing initially incorrect questions that became correct post-RLVR. In contrast to MATH-500, LiveMathBench shows no discernible improvement after RLVR, confirming its accuracy has no significant relationship with spurious RLVR.
• Figure 3: Perplexity Analysis With Accuracy. Full-text (top) and answer-only (bottom) perplexity heatmaps. Heatmaps display full-text and answer-only perplexity across checkpoints (step 0, 50, 100, 150) under spurious RLVR with incorrect rewards. Percentage annotations under each block show base model accuracy and accuracy improvement after RL training.
• Figure 4: The Perplexity Paradox. While answer-only perplexity decreases (orange), full-text perplexity increases (blue), suggesting a trade-off between memorization and general language modeling capability.
• Figure 5: Path Patching Accuracy Recovery Comparison. Left: Qwen exhibits a sustained peak at L18–L20 (marking the final injection of the correct answer) followed by a sudden drop at L21 (revealing a critical feature space divergence). Right: LLaMA shows no comparable recovery pattern and maintained extremely low recovery rates, confirming the absence of this memorization mechanism in the control model.