Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

TL;DR

This work tackles the coarse credit assignment of Group Relative Policy Optimization (GRPO) by introducing Outcome-grounded Advantage Reshaping (OAR), which redistributes sequence-level advantages to tokens based on their influence on the final answer. OAR provides two instantiations: OAR-P, a high-fidelity perturbation-based attribution, and OAR-G, a scalable gradient-based proxy, both integrated with a conservative Bi-Level reshaping that preserves total advantage mass. Empirical results on mathematical reasoning benchmarks show OAR substantially outperforms strong GRPO baselines, with OAR-P setting the upper bound and OAR-G delivering near-parity gains at a modest computational cost. This approach advances critic-free RL for LLM reasoning by aligning token-level credit with outcome relevance, improving stability and scalability for long-horizon tasks.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks. However, standard GRPO employs a coarse-grained credit assignment mechanism that propagates group-level rewards uniformly to to every token in a sequence, neglecting the varying contribution of individual reasoning steps. We address this limitation by introducing Outcome-grounded Advantage Reshaping (OAR), a fine-grained credit assignment mechanism that redistributes advantages based on how much each token influences the model's final answer. We instantiate OAR via two complementary strategies: (1) OAR-P, which estimates outcome sensitivity through counterfactual token perturbations, serving as a high-fidelity attribution signal; (2) OAR-G, which uses an input-gradient sensitivity proxy to approximate the influence signal with a single backward pass. These importance signals are integrated with a conservative Bi-Level advantage reshaping scheme that suppresses low-impact tokens and boosts pivotal ones while preserving the overall advantage mass. Empirical results on extensive mathematical reasoning benchmarks demonstrate that while OAR-P sets the performance upper bound, OAR-G achieves comparable gains with negligible computational overhead, both significantly outperforming a strong GRPO baseline, pushing the boundaries of critic-free LLM reasoning.

Figures (8)

• Figure 1: From broadcast credit to token reallocation.
• Figure 2: Overall architecture of the proposed OAR framework integrated into GRPO.
• Figure 3: Training dynamics on Qwen2.5-7B-Base(Top Row) and Qwen2.5-Math-7B(Bottom Row).
• Figure 4: Token-importance visualization on a reasoning trace: OAR-P (top) vs. Oracle causal mask (bottom).
• Figure 5: Causal-token recall under the counterfactual Oracle as a function of the top-$K\%$ important tokens.