Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards

TL;DR

The paper investigates how sample polarity shapes reinforcement learning with verifiable rewards (RLVR) for large reasoning models, showing that positive samples sharpen existing correct reasoning while negative samples promote exploration of new paths; both polarities are essential for robust RLVR. It analyzes training dynamics across multiple base LLMs and introduces adaptive and asymmetric token-level advantage shaping (A3PO) to allocate advantages more precisely across tokens and polarities. A3PO, built on the DAPO framework, yields superior performance and more stable training across five reasoning benchmarks and three LLMs. The work advances RLVR by revealing how granular advantage shaping interacts with polarity to balance exploration and exploitation, with practical implications for improving reasoning quality and generalization in verifiable reasoning settings.

Abstract

Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach.

Figures (31)

• Figure 1: RLVR training dynamics under three training paradigms on Deepseek-R1-Distilled-Qwen-7B.
• Figure 2: RLVR training reward across different training paradigms and base LLMs.
• Figure 3: Training behaviors of different paradigms.
• Figure 4: Polarity-level advantage shaping results on Qwen2.5-7B-Math. Each label is formatted as "PXNY", where "X" and "Y" represent the advantage scaling factors for positive and negative samples. For example, "P1N5" denotes positive sample weight $\times$1 and negative sample weight $\times$5.
• Figure 5: Token-level entropy-based advantage shaping. The x-axis indicates the entropy of shaped tokens (right: high entropy "H"; left: low entropy "L"). The y-axis shows shaped token polarity (top: positive "P"; bottom: negative "N"). Each label follows the format [Polarity][Entropy][Scaling Factor], where the first letter denotes token polarity, the second indicates entropy level, and the numeric value specifies the scaling factor applied to the advantage of those tokens. In the figure, lines with darker colors correspond to amplifying the advantage values of these tokens, while lighter colors indicate reducing their advantage values.