Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph

Tiezheng Guo, Qingwen Yang, Chen Wang, Yanyi Liu, Pan Li, Jiawei Tang, Dapeng Li, Yingyou Wen

TL;DR

This work addresses the knowledge limitations and hallucinations of large language models in knowledge graph question answering (KGQA). It introduces KnowledgeNavigator, a three-stage framework (Question Analysis, Knowledge Retrieval, Reasoning) that retrieves and structures external KG knowledge to guide LLM reasoning, while remaining plug-in compatible with any KG and LLM. Key contributions include a hop-depth predictor, LLM-guided iterative relation filtering, and triplet aggregation into natural-language prompts, enabling effective reasoning without frequent LLM retraining. Experimental results on MetaQA and WebQSP show substantial gains over prior KG-enhanced LLM methods and competitive performance with fully supervised models, highlighting the impact of structured external knowledge and careful prompt design on long, knowledge-intensive reasoning tasks.

Abstract

Large language model (LLM) has achieved outstanding performance on various downstream tasks with its powerful natural language understanding and zero-shot capability, but LLM still suffers from knowledge limitation. Especially in scenarios that require long logical chains or complex reasoning, the hallucination and knowledge limitation of LLM limit its performance in question answering (QA). In this paper, we propose a novel framework KnowledgeNavigator to address these challenges by efficiently and accurately retrieving external knowledge from knowledge graph and using it as a key factor to enhance LLM reasoning. Specifically, KnowledgeNavigator first mines and enhances the potential constraints of the given question to guide the reasoning. Then it retrieves and filters external knowledge that supports answering through iterative reasoning on knowledge graph with the guidance of LLM and the question. Finally, KnowledgeNavigator constructs the structured knowledge into effective prompts that are friendly to LLM to help its reasoning. We evaluate KnowledgeNavigator on multiple public KGQA benchmarks, the experiments show the framework has great effectiveness and generalization, outperforming previous knowledge graph enhanced LLM methods and is comparable to the fully supervised models.

KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph

TL;DR

This work addresses the knowledge limitations and hallucinations of large language models in knowledge graph question answering (KGQA). It introduces KnowledgeNavigator, a three-stage framework (Question Analysis, Knowledge Retrieval, Reasoning) that retrieves and structures external KG knowledge to guide LLM reasoning, while remaining plug-in compatible with any KG and LLM. Key contributions include a hop-depth predictor, LLM-guided iterative relation filtering, and triplet aggregation into natural-language prompts, enabling effective reasoning without frequent LLM retraining. Experimental results on MetaQA and WebQSP show substantial gains over prior KG-enhanced LLM methods and competitive performance with fully supervised models, highlighting the impact of structured external knowledge and careful prompt design on long, knowledge-intensive reasoning tasks.

Abstract

Large language model (LLM) has achieved outstanding performance on various downstream tasks with its powerful natural language understanding and zero-shot capability, but LLM still suffers from knowledge limitation. Especially in scenarios that require long logical chains or complex reasoning, the hallucination and knowledge limitation of LLM limit its performance in question answering (QA). In this paper, we propose a novel framework KnowledgeNavigator to address these challenges by efficiently and accurately retrieving external knowledge from knowledge graph and using it as a key factor to enhance LLM reasoning. Specifically, KnowledgeNavigator first mines and enhances the potential constraints of the given question to guide the reasoning. Then it retrieves and filters external knowledge that supports answering through iterative reasoning on knowledge graph with the guidance of LLM and the question. Finally, KnowledgeNavigator constructs the structured knowledge into effective prompts that are friendly to LLM to help its reasoning. We evaluate KnowledgeNavigator on multiple public KGQA benchmarks, the experiments show the framework has great effectiveness and generalization, outperforming previous knowledge graph enhanced LLM methods and is comparable to the fully supervised models.
Paper Structure (20 sections, 6 equations, 2 figures, 4 tables)

This paper contains 20 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Reasoning for KGQA
  4. Knowledge Graph Enhanced LLM
  5. Method
  6. Question Analysis
  7. Knowledge Retrieval
  8. Reasoning
  9. Experiments
  10. Dataset
  11. Baselines
  12. Experiment Details
  13. Main Results
  14. Ablation Study
  15. Impact of Number of Similar Questions
  16. ...and 5 more sections

Figures (2)

  • Figure 1: LLM suffers from hallucination and knowledge limitation, which can be solved with external knowledge.
  • Figure 2: An overview of KnowledgeNavigator. The framework consists of three consecutive phases: Question Analysis, Knowledge Retrieval, and Reasoning. The given example comes from MetaQA, describing a 2-hop reasoning task starting from "Babaloo Mandel" and ending with entities including "Tom Hanks". In the knowledge graph, solid lines indicate that entities or relations are retrieved as reasoning knowledge, while dashed lines indicate that entities or relations are discarded.