Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards

TL;DR

This paper identifies and addresses the shortcomings of outcome-based reinforcement learning for deep search agents by introducing CaRR, a fine-grained, citation-aware rubric rewards framework. CaRR decomposes complex questions into verifiable single-hop rubrics and evaluates agent trajectories along hidden-entity identification, citation grounding, and evidence connectivity. Building on CaRR, the authors propose C-GRPO, a mixed-reward RL algorithm that combines rubric rewards with outcome rewards to train more robust, comprehensively reasoned agents. Across 4B and 30B model scales on multiple benchmarks, C-GRPO achieves superior performance, reduces shortcut exploitation, and generalizes to open-ended deep research tasks, while maintaining favorable test-time scalability.

Abstract

Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents. However, existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process, and often lead to undesirable behaviors such as shortcut exploitation and hallucinations. To address these limitations, we propose \textbf{Citation-aware Rubric Rewards (CaRR)}, a fine-grained reward framework for deep search agents that emphasizes reasoning comprehensiveness, factual grounding, and evidence connectivity. CaRR decomposes complex questions into verifiable single-hop rubrics and requires agents to satisfy these rubrics by explicitly identifying hidden entities, supporting them with correct citations, and constructing complete evidence chains that link to the predicted answer. We further introduce \textbf{Citation-aware Group Relative Policy Optimization (C-GRPO)}, which combines CaRR and outcome rewards for training robust deep search agents. Experiments show that C-GRPO consistently outperforms standard outcome-based RL baselines across multiple deep search benchmarks. Our analysis also validates that C-GRPO effectively discourages shortcut exploitation, promotes comprehensive, evidence-grounded reasoning, and exhibits strong generalization to open-ended deep research tasks. Our code and data are available at https://github.com/THUDM/CaRR.

Figures (14)

• Figure 1: Pure outcome rewards fail to capture shortcut exploitation and hallucinations of deep search agents.
• Figure 2: Overview of (a) rubric initialization; (b) computation of context-aware rubric rewards; (c) C-GRPO.
• Figure 3: Left: Accuracy improvements by GRPO and C-GRPO over SFT models at 64k context length. Middle and Right: Test-time scaling performance of different models with respect to context budget and tool call budget.
• Figure 4: Training dynamics of GRPO and C-GRPO, including the changes of average tool call steps, outcome rewards, and rubric rewards.
• Figure 5: Description of search, open, and find tool.