Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning

TL;DR

The paper addresses the challenge of long-context reasoning by identifying evidence retrieval as a critical bottleneck and proposing Evidence-Augmented Reasoning (EAR) and Evidence-Augmented Policy Optimization (EAPO). EAPO replaces sparse outcome rewards with dense, process-level supervision via a Reward Model that computes Group-Relative Evidence Rewards, and couples this with Adaptive Reward-Policy Co-Evolution to keep supervision aligned with the evolving policy. Built on Group Relative Policy Optimization (GRPO), EAPO integrates multi-granular rewards ($R_f$, $R_e$, $R_a$) into the total reward $R_{total} = \alpha R_f + \beta R_e + \gamma R_a$, guiding high-quality evidence retrieval and reasoning. Across eight long-context benchmarks, EAPO achieves robust gains over state-of-the-art baselines, including larger proprietary models, demonstrating the value of dense, evidence-grounded supervision for trustworthy, precise reasoning in ultra-long contexts.

Abstract

While Reinforcement Learning (RL) has advanced LLM reasoning, applying it to long-context scenarios is hindered by sparsity of outcome rewards. This limitation fails to penalize ungrounded "lucky guesses," leaving the critical process of needle-in-a-haystack evidence retrieval largely unsupervised. To address this, we propose EAPO (Evidence-Augmented Policy Optimization). We first establish the Evidence-Augmented Reasoning paradigm, validating via Tree-Structured Evidence Sampling that precise evidence extraction is the decisive bottleneck for long-context reasoning. Guided by this insight, EAPO introduces a specialized RL algorithm where a reward model computes a Group-Relative Evidence Reward, providing dense process supervision to explicitly improve evidence quality. To sustain accurate supervision throughout training, we further incorporate an Adaptive Reward-Policy Co-Evolution mechanism. This mechanism iteratively refines the reward model using outcome-consistent rollouts, sharpening its discriminative capability to ensure precise process guidance. Comprehensive evaluations across eight benchmarks demonstrate that EAPO significantly enhances long-context reasoning performance compared to SOTA baselines.

Figures (10)

• Figure 1: Challenges in reasoning over extremely long contexts. The examples illustrate critical failure modes, including answers with no evidence support, incomplete or error retrieval. Our work explicitly targets this bottleneck to ensure high-fidelity evidence extraction.
• Figure 2: Overview of Tree-Structured Evidence Sampling. The process samples diverse evidence and reasoning paths, evaluates their scores, and derives the final answer.
• Figure 3: Performance analysis of Qwen3-30B-A3B-Instruct on Musique in LongBench. We compare Direct Reasoning, EAR variants with different evaluators, and an Oracle upper bound (Evidence-Grounded).
• Figure 4: Overview of EAPO. Left: The Evidence-Augmented Policy Optimization phase, utilizing dense evidence scores to guide the high-quality evidence extraction process. Right: The Adaptive Reward-Policy Co-Evolution cycle, which iteratively refines the reward model to sharpen its discriminative capability for accurate supervision.
• Figure 5: Task performance on the MuSiQue in LongBench. EAPO achieves superior convergence compared to the static reward baseline and the outcome-only GRPO.