Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping
Miao Peng, Weizhou Shen, Nuo Chen, Chenliang Li, Ming Yan, Jia Li
TL;DR
Long-context reasoning remains difficult for RL-based supervision due to sparse final rewards that overlook useful intermediate steps, creating an almost-there failure pattern. The authors address this with DeepReasonQA, a KG-driven synthesis framework that produces high-density, multi-hop long-context QA pairs with explicit reasoning chains, and LongPAS, a process-advantage shaping method that provides fine-grained credit assignment along Validity and Relevance dimensions. Empirical results on FRAMES, LongBench V2, and multi-hop QA tasks show that DeepReasonQA plus LongPAS outperforms strong RLVR baselines across diverse LLM backbones and approaches frontier models with far fewer parameters, while promoting more precise grounding and longer-range reasoning. This work demonstrates stable RL training and robust long-context reasoning, offering a scalable pathway to sophisticated reasoning agents capable of handling vast information environments.
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs short-context reasoning, but its performance degrades in long-context scenarios that require both precise grounding and robust long-range reasoning. We identify the "almost-there" phenomenon in long-context reasoning, where trajectories are largely correct but fail at the final step, and attribute this failure to two factors: (1) the lack of high reasoning density in long-context QA data that push LLMs beyond mere grounding toward sophisticated multi-hop reasoning; and (2) the loss of valuable learning signals during long-context RL training due to the indiscriminate penalization of partially correct trajectories with incorrect outcomes. To overcome this bottleneck, we propose DeepReasonQA, a KG-driven synthesis framework that controllably constructs high-difficulty, multi-hop long-context QA pairs with inherent reasoning chains. Building on this, we introduce Long-context Process Advantage Shaping (LongPAS), a simple yet effective method that performs fine-grained credit assignment by evaluating reasoning steps along Validity and Relevance dimensions, which captures critical learning signals from "almost-there" trajectories. Experiments on three long-context reasoning benchmarks show that our approach substantially outperforms RLVR baselines and matches frontier LLMs while using far fewer parameters. Further analysis confirms the effectiveness of our methods in strengthening long-context reasoning while maintaining stable RL training.