Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

Xujie Yuan, Yongxu Liu, Shimin Di, Shiwen Wu, Libin Zheng, Rui Meng, Lei Chen, Xiaofang Zhou, Jian Yin

TL;DR

This work analyzes when and how to use Knowledge Graphs within Retrieval Augmented Generation by introducing a configurable KG-RAG framework and two methods, Pilot and Meta, and evaluating 6 KG-RAG approaches across 9 datasets and 17 LLMs. It shows strong, task-dependent gains in domain-specific medical QA and exams, with more modest improvements in open-domain tasks and limited benefits in multi-turn dialogues. The study demonstrates that KG quality and configuration interactions critically affect performance, and that benefits persist across BOS-LLMs as model scales. Overall, KG-RAG can match or surpass commercial models in specialized domains, offering practical guidelines for deploying KG-augmented retrieval in real-world applications.

Abstract

The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 9 datasets in diverse domains and scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs, and combining Metacognition with KG-RAG as a pilot attempt. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.

A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

TL;DR

This work analyzes when and how to use Knowledge Graphs within Retrieval Augmented Generation by introducing a configurable KG-RAG framework and two methods, Pilot and Meta, and evaluating 6 KG-RAG approaches across 9 datasets and 17 LLMs. It shows strong, task-dependent gains in domain-specific medical QA and exams, with more modest improvements in open-domain tasks and limited benefits in multi-turn dialogues. The study demonstrates that KG quality and configuration interactions critically affect performance, and that benefits persist across BOS-LLMs as model scales. Overall, KG-RAG can match or surpass commercial models in specialized domains, offering practical guidelines for deploying KG-augmented retrieval in real-world applications.

Abstract

The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 9 datasets in diverse domains and scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs, and combining Metacognition with KG-RAG as a pilot attempt. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.

Paper Structure

This paper contains 25 sections, 2 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. The Effectiveness of KG-RAG Across Diverse Domains and Scenarios
  4. Can KG-RAG improve BOS-LLMs across different task domains and scenarios?
  5. More Studies on Generalizability and KG Quality
  6. Bridging the Gap: KG-RAG Enhanced BOS-LLMs vs. Commercial Models
  7. Can BOS-LLMs with KG-RAG are better than commercial LLMs?
  8. KG-RAG with Advanced BOS-LLMs
  9. Optimizing KG-RAG Performance: Component Configurations
  10. KG-RAG Configurations
  11. How effective are different KG-RAG configurations?
  12. Conclusion
  13. Complementary Experiments
  14. Datasets
  15. The remaining Experimental Results
  16. ...and 10 more sections

Figures (2)

  • Figure 1: The mind and pipeline flows of KG-RAG.
  • Figure 2: KG-RAG Methods and their Configurations.

Theorems & Definitions (1)

  • Remark 3.1