ProGraph-R1: Progress-aware Reinforcement Learning for Graph Retrieval Augmented Generation
Jinyoung Park, Sanghyeok Lee, Omar Zia Khan, Hyunwoo J. Kim, Joo-Kyung Kim
TL;DR
This work addresses hallucination and reasoning limits in knowledge-intensive QA by marrying graph-structured retrieval with progress-aware reinforcement learning. It introduces ProGraph-R1, which uses a structure-aware hypergraph retrieval that combines semantic relevance with graph connectivity, and a step-progress policy optimization that provides dense, intermediate rewards to guide multi-hop reasoning. The approach extends GRPO with step-level advantages and adds retrieval signals that reflect both informativeness and structural coherence, culminating in a total reward that blends outcome, progress, and structural terms. Empirical results on 2WikiMultihopQA, HotPotQA, MuSiQue, and Natural Questions show that ProGraph-R1 consistently outperforms Graph-R1 and other baselines, especially on multi-hop tasks, and ablations confirm the contributions of both retrieval and step-wise optimization. Overall, the method improves factual grounding and reasoning efficiency in GraphRAG setups, contributing to more reliable, graph-grounded generation in knowledge-intensive applications.
Abstract
Graph Retrieval-Augmented Generation (GraphRAG) has been successfully applied in various knowledge-intensive question answering tasks by organizing external knowledge into structured graphs of entities and relations. It enables large language models (LLMs) to perform complex reasoning beyond text-chunk retrieval. Recent works have employed reinforcement learning (RL) to train agentic GraphRAG frameworks that perform iterative interactions between LLMs and knowledge graphs. However, existing RL-based frameworks such as Graph-R1 suffer from two key limitations: (1) they primarily depend on semantic similarity for retrieval, often overlooking the underlying graph structure, and (2) they rely on sparse, outcome-level rewards, failing to capture the quality of intermediate retrieval steps and their dependencies. To address these limitations, we propose ProGraph-R1, a progress-aware agentic framework for graph-based retrieval and multi-step reasoning. ProGraph-R1 introduces a structure-aware hypergraph retrieval mechanism that jointly considers semantic relevance and graph connectivity, encouraging coherent traversal along multi-hop reasoning paths. We also design a progress-based step-wise policy optimization, which provides dense learning signals by modulating advantages according to intermediate reasoning progress within a graph, rather than relying solely on final outcomes. Experiments on multi-hop question answering benchmarks demonstrate that ProGraph-R1 consistently improves reasoning accuracy and generation quality over existing GraphRAG methods.