ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation

TL;DR

ProRAG tackles reward sparsity and process hallucinations in multi-hop retrieval-augmented generation by introducing a four-stage, process-supervised RL framework. It builds a Monte Carlo Tree Search–based Process Reward Model to provide dense step-level feedback, refines a policy with PRM-filtered trajectories, and then trains online with a dual-granularity advantage that blends step-level process rewards with global outcome signals. The approach delivers superior performance across five benchmarks, with strong data efficiency and low inference latency due to internalizing reasoning capabilities. These results demonstrate that dense, on-policy process supervision can significantly improve long-horizon reasoning and retrieval planning in real-world RAG deployments.

Abstract

Reinforcement learning (RL) has become a promising paradigm for optimizing Retrieval-Augmented Generation (RAG) in complex reasoning tasks. However, traditional outcome-based RL approaches often suffer from reward sparsity and inefficient credit assignment, as coarse-grained scalar rewards fail to identify specific erroneous steps within long-horizon trajectories. This ambiguity frequently leads to "process hallucinations", where models reach correct answers through flawed logic or redundant retrieval steps. Although recent process-aware approaches attempt to mitigate this via static preference learning or heuristic reward shaping, they often lack the on-policy exploration capabilities required to decouple step-level credit from global outcomes. To address these challenges, we propose ProRAG, a process-supervised reinforcement learning framework designed to integrate learned step-level supervision into the online optimization loop. Our framework consists of four stages: (1) Supervised Policy Warmup to initialize the model with a structured reasoning format; (2) construction of an MCTS-based Process Reward Model (PRM) to quantify intermediate reasoning quality; (3) PRM-Guided Reasoning Refinement to align the policy with fine-grained process preferences; and (4) Process-Supervised Reinforcement Learning with a dual-granularity advantage mechanism. By aggregating step-level process rewards with global outcome signals, ProRAG provides precise feedback for every action. Extensive experiments on five multi-hop reasoning benchmarks demonstrate that ProRAG achieves superior overall performance compared to strong outcome-based and process-aware RL baselines, particularly on complex long-horizon tasks, validating the effectiveness of fine-grained process supervision. The code and model are available at https://github.com/lilinwz/ProRAG.

Figures (6)

• Figure 1: Comparison of optimization paradigms. (a) Outcome-based RL suffers from spurious correlations due to sparse global signals. (b) Offline DPO is limited by distribution shifts caused by static datasets. (c) ProRAG (Ours) utilizes dense step-level supervision for precise credit assignment and dynamic error correction.
• Figure 2: The left panel presents the reasoning format of our model, while the right illustrates the Monte Carlo Tree Search (MCTS) mechanism used to train the Process Reward Model.
• Figure 3: The framework of Process-Supervised Reinforcement Learning.
• Figure 4: Training curves of ProRAG compared to the GRPO baseline and the unrefined variant (w/o Refinement).
• Figure 5: Performance comparison (F1 Score) on PopQA, HotpotQA, and Musique datasets with different numbers of retrieved documents ($k \in \{1, 3, 5\}$).