Probing RLVR training instability through the lens of objective-level hacking

TL;DR

This work addresses RLVR training instability in Mixture-of-Experts models by introducing objective-level hacking, a framework in which token-level credit misalignment creates system-level spurious signals in the optimization objective, denoted as $\Delta \mathcal{J}(\theta)$. It unifies several instability sources under a causal account and formalizes how training–inference discrepancy grows via off-policy effects, quantified by quantities like $\rho_{i,t}$ and covariances with $X_{i,t}(\theta)$. Through extensive experiments on a 30B MoE model, the authors demonstrate that biased token-level weight distortions can drive discrepancy growth and destabilize training, while unbiased variance-based perturbations do not, supporting a causal link. The results offer principled design guidance for stable RLVR algorithms and underscore the importance of controlling token-level credit biases, rollout–training mismatch, and numerical-precision artifacts in MoE RLVR systems.

Abstract

Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.

Figures (17)

• Figure 1: Illustration of the mechanism underlying training instability dynamics in RLVR. Numerical noise and token-level modulations introduce distortions in token-level weights, which are effectively equivalent to biased perturbations of the optimization objective. Such biases alter the optimization direction toward spurious signals, ultimately resulting in a growing training–inference discrepancy.
• Figure 2: Influence of initial training-inference discrepancy. The grey curve shows the training dynamics with token-level clipping, while the green curve corresponds to the same setting augmented with TIS correction. From left to right, the four panels show: (a) training reward, (b) validation accuracy, (c) training-inference discrepancy, and (d) token entropy during training.
• Figure 3: Token probabilities in inference mode vs. training mode. (a) and (b) subplots are taken from the token-level clipping with TIS correction at gradient steps 20 and 900 in Fig. \ref{['fig:ini disc']}, respectively. The red dashed line marks $y=x$, and the PCC between the two probabilities is annotated in the top-left corner.
• Figure 4: Influence of token-level clipping. Blue curves correspond to training dynamics with sequence-level clipping. Red curves represent token-level clipping with varying strengths, where darker shades indicate stronger clipping and lighter shades represent weaker clipping. In the legend, "strong", "mid", and "low" correspond to right clipping ranges of 0.2, 0.24, and 0.28, respectively.
• Figure 5: Influence of injected token-level weight distortion. Blue curves correspond to training dynamics with sequence-level clipping. Purple curves denote the same configuration with different strengths of injected token-level weight distortion, where darker to lighter shades indicate stronger to weaker one, respectively. $\delta$ in the legend indicates the distortion strength defined in Eq. (\ref{['eq: distortion strength']}).

Theorems & Definitions (1)

• proof : Proof