Rewards as Labels: Revisiting RLVR from a Classification Perspective

TL;DR

The paper identifies gradient misassignment and domination in GRPO-style RLVR methods, which hinder efficient credit assignment for reasoning tasks. It proposes REAL, a classification-based framework that treats verifiable rewards as categorical labels and introduces anchor logits to produce monotonic, bounded gradient weights. Empirical results across AIME, MATH, AMC, Minerva, and Olympiad Bench show REAL outperforms GRPO, DAPO, and GSPO on 1.5B and 7B models, with improved training stability and even competitive performance under BCE. By linking RLVR with a principled classification objective, REAL offers a scalable, stable approach for improving large language models on complex, rule-based reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks by providing explicit rule-based supervision. Among RLVR methods, GRPO and its variants have achieved strong empirical performance. Despite their success, we identify that they suffer from Gradient Misassignment in Positives and Gradient Domination in Negatives, which lead to inefficient and suboptimal policy updates. To address these issues, we propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights, thereby reformulating policy optimization as a classification problem. Building on this, we further introduce anchor logits to enhance policy learning. Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts and effectively mitigating the identified mismatches. Extensive experiments on mathematical reasoning benchmarks show that REAL improves training stability and consistently outperforms GRPO and strong variants such as DAPO. On the 1.5B model, REAL improves average Pass@1 over DAPO by 6.7%. These gains further scale to 7B model, REAL continues to outperform DAPO and GSPO by 6.2% and 1.7%, respectively. Notably, even with a vanilla binary cross-entropy, REAL remains stable and exceeds DAPO by 4.5% on average.

Figures (5)

• Figure 1: Gradient magnitude visualizations in GRPO and proposed REAL. We visualize the gradient magnitude $|\mathcal{W}_\text{GRPO}|$ against relative log-probability $s_t$ (shown in red) and $|\mathcal{W}_\text{REAL}|$ against length-normalized relative log probability score $\bar{s}$ (shown in blue) . (a) Positive samples ($r=1$): GRPO suffers from Gradient Misassignment. In contrast, the gradient magnitude of our proposed REAL monotonically decreases with increasing relative log-probability. (b) Negative samples ($r=0$): GRPO exhibits Gradient Domination, whereas REAL enforces strictly bounded gradients, ensuring stable training. For the curves shown in the figure, GRPO uses clipping $\varepsilon = 0.2$, and $A = 1$ as defined in Eq \ref{['eq:grpo gradient']}, while REAL uses temperature $\tau = 0.5$ and $C_+ = C_- = 4$ as defined in Eq \ref{['eq:real gradient']}.
• Figure 2: Importance of hard tokens in GRPO. We show the fraction of tokens falling into different intervals at the 200th step of GRPO training. Most tokens lie within the region $[1-\varepsilon,1+\varepsilon]$. We compare GRPO with vanilla clipping (clip positives when $\rho_t>1+\varepsilon$ and negatives when $\rho_t<1-\varepsilon$) against a bidirectional variant that discards both positive and negative tokens with $\rho_t\notin[1-\varepsilon,1+\varepsilon]$. Although bidirectional clipping removes an additional $\sim$0.3% of tokens, validation Pass@1 drops sharply, underscoring the importance of these hard tokens in positive and negative samples.
• Figure 3: Overview of our REAL framework. REAL formulates RLVR as a classification problem by treating rewards as explicit labels. $s$ denotes the prediction score.
• Figure 4: Training dynamics of the DeepSeek-R1-Distill-Qwen-1.5B model. We compare REAL against three baselines—GRPO, DAPO, and GSPO—across four key metrics: Entropy, Validation Score (Pass@1 on AIME 2024, averaged over 8 samples), Reward, and 100% Solved Ratio.
• Figure 5: Further analysis on BCE loss and Anchor Logits. Training dynamics and validation Pass@1 on AIME 2024 of different methods: DAPO, REAL w/ BCE Loss, REAL w/o Anchor Logits, and REAL (Ours).