Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning

TL;DR

MemQ addresses the leakage of reasoning into tool invocation in KGQA by introducing a memory-augmented framework that stores natural-language descriptions of query statements. It separates memory construction, NL-based reasoning, and query reconstruction, enabling readable step-by-step reasoning and more faithful SPARQL query reconstruction. Experimental results on WebQSP and CWQ show state-of-the-art performance, improved reasoning quality (lower $GoldGED$, higher $EHR$), and strong data efficiency and model universality. The approach offers a practical, interpretable path for reliable LLM-based KGQA with potential for plug-and-play across tools and knowledge bases.

Abstract

Large language models (LLMs) have achieved remarkable performance on knowledge graph question answering (KGQA) tasks by planning and interacting with knowledge graphs. However, existing methods often confuse tool utilization with knowledge reasoning, harming readability of model outputs and giving rise to hallucinatory tool invocations, which hinder the advancement of KGQA. To address this issue, we propose Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning (MemQ) to decouple LLM from tool invocation tasks using LLM-built query memory. By establishing a memory module with explicit descriptions of query statements, the proposed MemQ facilitates the KGQA process with natural language reasoning and memory-augmented query reconstruction. Meanwhile, we design an effective and readable reasoning to enhance the LLM's reasoning capability in KGQA. Experimental results that MemQ achieves state-of-the-art performance on widely used benchmarks WebQSP and CWQ.

Figures (6)

• Figure 1: Comparing reasoning methods designed with knowledge graph query tools with proposed memory-augmented method MemQ.
• Figure 2: The overall framework of MemQ. During the memory construction stage, we describe the question with its query history using the LLMs to get the reasoning steps. In the inference stage, we reconstruct the query using the recalled query sentences based on the reasoning results.
• Figure 3: Here we present the illustration of the 3 distinct graph structures.
• Figure 4: Case of MemQ, we retrieve memories based on the reasoning steps and reconstruct the final query.
• Figure 5: We evaluate the Hits@1 and F1 scores of the LLaMA-3 Reasoning LLM across varying proportions of training data.