EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering

TL;DR

EPERM addresses KGQA by integrating LLM reasoning with KG structure through a probabilistic graphical-model formulation, decomposing the task into subgraph retrieval, evidence-path generation with weights, and answer prediction. It introduces a three-stage EPERM framework and jointly tunes components via an ELBO-based objective, emphasizing the varying importance of different KG structures through weighted evidence paths. The approach achieves state-of-the-art or competitive results on WebQSP and CWQ, with clear gains in Hits@1 and F1, and demonstrates robustness via ablations and hyperparameter analyses. The work offers a practical, unified method to reduce hallucinations and improve reasoning fidelity in KGQA using instruction-tuned LLMs.

Abstract

Due to the remarkable reasoning ability, Large language models (LLMs) have demonstrated impressive performance in knowledge graph question answering (KGQA) tasks, which find answers to natural language questions over knowledge graphs (KGs). To alleviate the hallucinations and lack of knowledge issues of LLMs, existing methods often retrieve the question-related information from KGs to enrich the input context. However, most methods focus on retrieving the relevant information while ignoring the importance of different types of knowledge in reasoning, which degrades their performance. To this end, this paper reformulates the KGQA problem as a graphical model and proposes a three-stage framework named the Evidence Path Enhanced Reasoning Model (EPERM) for KGQA. In the first stage, EPERM uses the fine-tuned LLM to retrieve a subgraph related to the question from the original knowledge graph. In the second stage, EPERM filters out the evidence paths that faithfully support the reasoning of the questions, and score their importance in reasoning. Finally, EPERM uses the weighted evidence paths to reason the final answer. Since considering the importance of different structural information in KGs for reasoning, EPERM can improve the reasoning ability of LLMs in KGQA tasks. Extensive experiments on benchmark datasets demonstrate that EPERM achieves superior performances in KGQA tasks.

Figures (5)

• Figure 1: The directed graphical model of KGQA tasks.
• Figure 2: Overview of the proposed EPERM framework. The subgraph retriever module aims to retrieve the question-related subgraph. The evidence path finder module aims to find and score the importance of evidence reasoning paths. The answer predictor module aims to reason the final answer based on the weighted evidence paths.
• Figure 3: The Hit@1 scores of EPERM with the total number of generated plans $s$ and the number of Top-$S$ path filtered in every hop.
• Figure 4: Example of EPERM reasoning based on weighted evidence paths.
• Figure 5: The directed graphical model of KGQA tasks.