Discovering Process-Outcome Credit in Multi-Step LLM Reasoning

TL;DR

The paper tackles sparse outcome rewards in RL-based LLM reasoning by introducing Step-wise Marginal Information Gain (MIG) evaluated against a Monotonic Historical Watermark (HWM) to credit informative intermediate steps. A Decoupled Masking strategy combines MIG-driven process exploration with outcome-grounded training via an Outcome loss and Dual-Gated SFT for stable data distillation. Across eight textual and multimodal benchmarks, the approach yields superior sample efficiency, higher final accuracy, and strong out-of-distribution robustness compared to GRPO, while mitigating reward hacking. The method demonstrates modality-agnostic improvements and suggests a scalable path for enhancing multi-step reasoning in larger LLMs without relying on external expert annotations.

Abstract

Reinforcement Learning (RL) serves as a potent paradigm for enhancing reasoning capabilities in Large Language Models (LLMs), yet standard outcome-based approaches often suffer from reward sparsity and inefficient credit assignment. In this paper, we propose a novel framework designed to provide continuous reward signals, which introduces a Step-wise Marginal Information Gain (MIG) mechanism that quantifies the intrinsic value of reasoning steps against a Monotonic Historical Watermark, effectively filtering out training noise. To ensure disentangled credit distribution, we implement a Decoupled Masking Strategy, applying process-oriented rewards specifically to the chain-of-thought (CoT) and outcome-oriented rewards to the full completion. Additionally, we incorporate a Dual-Gated SFT objective to stabilize training with high-quality structural and factual signals. Extensive experiments across textual and multi-modal benchmarks (e.g., MATH, Super-CLEVR) demonstrate that our approach consistently outperforms baselines such as GRPO in both sample efficiency and final accuracy. Furthermore, our model exhibits superior out-of-distribution robustness, demonstrating promising zero-shot transfer capabilities to unseen and challenging reasoning tasks.

Figures (5)

• Figure 1: Overview of the proposed Framework. Our approach synergizes intrinsic exploration with strict outcome alignment through three key mechanisms: (Left) Group Sampling: For each prompt, we sample a group of trajectories and parse them into discrete steps using strict structural tags. The exact prompt template and a concrete case study of the step-wise generation are detailed in Appendix \ref{['app:prompt_details']}. (Middle) Step-wise Marginal Information Gain (MIG): We calculate the dense reward $g_k$ as the rectified semantic breakthrough against a Monotonic Historical Watermark ($h_{k-1}$), strictly rewarding only non-monotonic logical discoveries. (Right) Decoupled Hybrid Optimization: The final objective combines the dense signal (for process exploration) and sparse correctness feedback (for outcome constraint) via a decoupled masking strategy, ensuring that intrinsic curiosity operates strictly within the bounds of correctness.
• Figure 2: Training Dynamics across Textual (Top) and Multimodal (Bottom) Benchmarks. Plots show the training accuracy on rollout over steps. Faint lines indicate raw data, while solid lines denote Gaussian-smoothed trends. Red curves (Ours) consistently exhibit a steeper initial ascent compared to GRPO (Blue), indicating superior sample efficiency. Notably, in structured reasoning tasks like GSM8K and Super-CLEVR, our Step-wise MIG reward enables the policy to discover valid reasoning paths significantly faster than the outcome-sparse baseline.
• Figure 3: Ablation Analysis on Average Performance. We compare the average accuracy across four benchmarks (MATH, CSQA, SVAMP, AIME). While removing the SFT component ("w/o SFT") causes only minor degradation on standard tasks, it leads to a catastrophic failure on the hardest benchmark (AIME), dropping the average performance significantly. This confirms that Dual-Gated SFT is essential for stabilizing complex reasoning capabilities.
• Figure 4: Empirical Validation of the Reward Proxy.(a) Violin plots showing the distribution of MCTS-estimated values $\hat{V}$ across different log-likelihood buckets. We observe a strong monotonic correlation: as $\ell_k$ approaches 0 (higher confidence), both the median (white bar) and mean (red diamond) value increase sharply, confirming $\ell_k$ as a reliable dense signal. (b) Value distribution across reasoning stages. While late steps generally exhibit higher values, early steps show significant variance with a "long tail" of high-value states.
• Figure 5: Evolution of Accumulated Step-wise Rewards (HWM). The curve illustrates the average accumulated monotonic information gain per episode during training. MIG (Ours) demonstrates a consistent upward trajectory, confirming that the model learns to generate strictly positive information increments throughout the reasoning chain. In contrast, the outcome-based GRPO baseline fails to accumulate intermediate dense rewards, highlighting the sparsity of its supervision signal.