AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction

TL;DR

AutoGraph-R1 addresses the core gap between KG construction and downstream QA by introducing an end-to-end reinforcement learning framework that tunes an LLM-based KG constructor to maximize QA utility. It designs two task-aware rewards, the Knowledge-Carrying Reward $R_C$ and the Knowledge-Indexing Reward $R_I$, and optimizes the KG with Group-Relative Policy Optimization (GRPO) while keeping the retriever fixed. Across five QA benchmarks and multiple RAG paradigms, AutoGraph-R1 yields consistent end-to-end gains and improves both retrieval effectiveness and intrinsic factual quality of graphs, illustrating that functionally useful graphs can be learned directly from task signals. This work shifts the focus from building intrinsically good graphs to constructing graphs demonstrably useful for the intended downstream reasoning tasks, enabling more effective retrieval-augmented generation systems. $R_C$ and $R_I$ guide the KG construction to support multi-hop reasoning and efficient indexing, respectively, demonstrating robust improvements in both graph knowledge retrieval and passage recall.

Abstract

Building effective knowledge graphs (KGs) for Retrieval-Augmented Generation (RAG) is pivotal for advancing question answering (QA) systems. However, its effectiveness is hindered by a fundamental disconnect: the knowledge graph (KG) construction process is decoupled from its downstream application, yielding suboptimal graph structures. To bridge this gap, we introduce AutoGraph-R1, the first framework to directly optimize KG construction for task performance using Reinforcement Learning (RL). AutoGraph-R1 trains an LLM constructor by framing graph generation as a policy learning problem, where the reward is derived from the graph's functional utility in a RAG pipeline. We design two novel, task-aware reward functions, one for graphs as knowledge carriers and another as knowledge indices. Across multiple QA benchmarks, AutoGraph-R1 consistently enables graph RAG methods to achieve significant performance gains over using task-agnostic baseline graphs. Our work shows it is possible to close the loop between construction and application, shifting the paradigm from building intrinsically ``good'' graphs to building demonstrably ``useful'' ones.

Figures (13)

• Figure 1: Bridging the Disconnection Between KG Construction and Utility. (A) A KG optimized for intrinsic metrics can be too fragmented for multi-hop QA, causing retrieval to fail. (B) AutoGraph-R1 uses an end-task RL reward to build a functionally superior graph, ensuring its structure is optimized to support various graph retrievers for successful reasoning.
• Figure 2: Overview of the AutoGraph-R1 Framework. AutoGraph-R1 optimizes knowledge graph construction for downstream utility using reinforcement learning. During the training phase (left), a graph constructor is fine-tuned with GRPO. The reward signal is derived from the performance of a graph retriever on the generated KG, directly measuring the graph's functional quality. During the inference phase (right), the trained constructor is used to build a large-scale KG from a general corpus, which then serves a downstream graph-based RAG system.
• Figure 3: Effective Training Dynamics with the Deducible Reward ($R_C$). Training curves for the Graph Knowledge Retriever setting. (a) The reward, measuring answer deducibility, steadily increases and converges, demonstrating the policy is successfully learning its objective. (b) The advantage gain trends towards a small negative value, indicating that the value function's estimate of expected reward is rising quickly while the policy makes stable, incremental improvements. This dynamic, coupled with the rising absolute reward, points to effective and controlled optimization.
• Figure 4: Effective Training Dynamics with the Knowledge-Indexing Reward ($R_I$). Training curves for the Graph-based Text Retriever setting. (a) The reward, measuring passage recall, shows a clear upward trend of improvement. (b) The advantage gain dynamic, paired with the rising reward curve, confirms that the policy is effectively learning from this stable, task-specific signal.
• Figure 5: Unstable Training Dynamics Using a Naive F1 Reward. Training curves for the ablation study where the final RAG F1 score is used as the reward. (a) The F1 reward signal is highly volatile and shows no clear upward trend, providing a noisy and ineffective learning signal. (b) Consequently, the advantage gain remains flat and centered around zero, confirming that the policy is failing to find a consistent direction for improvement. This leads to stalled optimization, as reflected in the poor downstream results.