Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs

TL;DR

Graph-RFT tackles knowledge-graph question answering under incomplete knowledge graphs by enabling autonomous planning and adaptive retrieval across KG search and web search. It introduces a two-stage learning framework: first, chain-of-thought fine-tuning to activate planning and reasoning and resolve the cold-start problem, and second, plan–retrieval guided reinforcement learning with a multi-reward objective that coordinates KG and web retrieval under incomplete KG conditions. The method employs a Cartesian-inspired planning module and logical constraints to ensure globally coherent multi-step reasoning, and integrates explicit planning steps with retrieval actions via a structured template. Empirical results across CWQ, GrailQA, WebQSP, and SimpleQuestions demonstrate state-of-the-art accuracy, robustness to KG incompleteness, and clear ablations showing the contributions of SFT and the multi-reward design. Overall, Graph-RFT offers a practical framework for robust KGQA in real-world settings with missing or partial graph coverage.

Abstract

Knowledge Graph Question Answering aims to answer natural language questions by reasoning over structured knowledge graphs. While large language models have advanced KGQA through their strong reasoning capabilities, existing methods continue to struggle to fully exploit both the rich knowledge encoded in KGs and the reasoning capabilities of LLMs, particularly in complex scenarios. They often assume complete KG coverage and lack mechanisms to judge when external information is needed, and their reasoning remains locally myopic, failing to maintain coherent multi-step planning, leading to reasoning failures even when relevant knowledge exists. We propose Graph-RFT, a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm, that enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions. Graph-RFT introduces a chain-of-thought fine-tuning method with a customized plan-retrieval dataset activates structured reasoning and resolves the GRPO cold-start problem. It then introduces a novel plan-retrieval guided reinforcement learning process integrates explicit planning and retrieval actions with a multi-reward design, enabling coverage-aware retrieval scheduling. It employs a Cartesian-inspired planning module to decompose complex questions into ordered subquestions, and logical expression to guide tool invocation for globally consistent multi-step reasoning. This reasoning retrieval process is optimized with a multi-reward combining outcome and retrieval specific signals, enabling the model to learn when and how to combine KG and web retrieval effectively.

Figures (8)

• Figure 1: The comparison of Semantic Parsing, Retrieval-Augmented Methods, and Graph-RFT.
• Figure 2: Graph-RFT Reasoning Trajectory for SFT dataset construction and rollout process for RL.
• Figure 3: Graph-RFT Training Framework: SFT-based Reasoning Activation (S1) and RL-based Reasoning Enhancement (S2).
• Figure 4: Ratio of web search operation in different KG settings under rewards with penalty or not.
• Figure 5: Performance of GoG and our model under different KG settings.