From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning

TL;DR

This work tackles the instability and sparsity of absolute reward signals in group-based reinforcement learning for LLMs. It proposes Reinforcement Learning with Relative Rewards (RLRR), which replaces scalar rewards with intra-group relative rankings, supported by the Ranking Reward Model (Ranking RM) that outputs listwise rankings. RLRR combines HRR or PRR with a hierarchical re-ranking mechanism to ensure correctness and conciseness while bounding reward variance for stability. Empirical results across verifiable reasoning and open-ended writing tasks show consistent improvements over GRPO baselines, and the Ranking RM provides robust, data-efficient guidance that generalizes across domains. Overall, relative ranking signals enable more reliable and efficient learning in group-based RL scenarios for LLMs.

Abstract

Reinforcement learning has become a cornerstone for enhancing the reasoning capabilities of Large Language Models, where group-based approaches such as GRPO have emerged as efficient paradigms that optimize policies by leveraging intra-group performance differences. However, these methods typically rely on absolute numerical rewards, introducing intrinsic limitations. In verifiable tasks, identical group evaluations often result in sparse supervision, while in open-ended scenarios, the score range instability of reward models undermines advantage estimation based on group means. To address these limitations, we propose Reinforcement Learning with Relative Rewards (RLRR), a framework that shifts reward shaping from absolute scoring to relative ranking. Complementing this framework, we introduce the Ranking Reward Model, a listwise preference model tailored for group-based optimization to directly generate relative rankings. By transforming raw evaluations into robust relative signals, RLRR effectively mitigates signal sparsity and reward instability. Experimental results demonstrate that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.

Figures (13)

• Figure 1: Comparative analysis of reward formulations. (a) Sparse rewards cause zero intra-group variance, reducing data utilization efficiency during GRPO training. (b) Scalar reward models exhibit higher numerical dispersion, compromising stability in intra-group advantage estimation. (c) RLRR utilizes relative ranking to sustain stable advantages through consistent gradient signals.
• Figure 2: Overview of RLRR. The framework derives intra-group preference rankings based on Reward Model outputs, while incorporating correctness and length constraints. It then integrates relative rewards using either HRR or PRR, contingent on the availability of rule-based correctness rewards. Finally, the advantage estimation process accounts for correctness consistency and applies a clipping mechanism to handle contradictory samples.
• Figure 3: Training-data processing pipeline for the Ranking Reward Model.
• Figure 4: Reward-Guided Best-of-N Test-Time Scaling for Enhanced Inference Performance.
• Figure 5: Comparison of GRPO and RLRR Across Different Data Difficulty Levels.