Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning

TL;DR

This work addresses the limitations of outcome-based reinforcement learning in agentic RAG by introducing process-supervised RL with ReasonRAG, SPRE, and Monte Carlo Tree Search. It automatically constructs RAG-ProGuide, a 5k-question process-level dataset with 13,289 preference pairs, to guide fine-grained policy optimization via Direct Preference Optimization. Across five QA benchmarks, ReasonRAG substantially outperforms outcome-based baselines while using dramatically less training data (5k vs 90k queries), and shows strong multi-hop and out-of-domain capabilities. The approach highlights the practical impact of dense, process-level rewards for improving exploration, efficiency, and generalization in autonomous LLM reasoning with external knowledge sources.

Abstract

Retrieval-augmented generation (RAG) enhances the text generation capabilities of large language models (LLMs) by integrating external knowledge and up-to-date information. However, traditional RAG systems are limited by static workflows and lack the adaptability required for multistep reasoning and complex task management. To address these limitations, agentic RAG systems (e.g., DeepResearch) have been proposed, enabling dynamic retrieval strategies, iterative context refinement, and adaptive workflows for handling complex search queries beyond the capabilities of conventional RAG. Recent advances, such as Search-R1, have demonstrated promising gains using outcome-based reinforcement learning, where the correctness of the final answer serves as the reward signal. Nevertheless, such outcome-supervised agentic RAG methods face challenges including low exploration efficiency, gradient conflict, and sparse reward signals. To overcome these challenges, we propose to utilize fine-grained, process-level rewards to improve training stability, reduce computational costs, and enhance efficiency. Specifically, we introduce a novel method ReasonRAG that automatically constructs RAG-ProGuide, a high-quality dataset providing process-level rewards for (i) query generation, (ii) evidence extraction, and (iii) answer generation, thereby enhancing model inherent capabilities via process-supervised reinforcement learning. With the process-level policy optimization, the proposed framework empowers LLMs to autonomously invoke search, generate queries, extract relevant evidence, and produce final answers. Compared to existing approaches such as Search-R1 and traditional RAG systems, ReasonRAG, leveraging RAG-ProGuide, achieves superior performance on five benchmark datasets using only 5k training instances, significantly fewer than the 90k training instances required by Search-R1.

Figures (25)

• Figure 1: Outcome-supervised vs. process-supervised RL for multi-step reasoning. Each circle denotes one-step reasoning response, where the correct response is colored as "green" and error response is colored as "red".
• Figure 1: Overall Dataset Statistics
• Figure 2: Framework of ReasonRAG. Figure (a) illustrates the policy optimization based on process supervision. MCTS guides the construction of the state-action tree and the assignment of process-level rewards for fine-grained policy optimization. (Actions derived from the same state are color-coded by reward: green circle (highest), red circle (lowest).) Figure (b) demonstrates an inference example.
• Figure 3: Dataset Distribution.
• Figure 4: A Tree Example for the process-level preference annotations.