In-depth Analysis of Graph-based RAG in a Unified Framework

TL;DR

The paper addresses the fragmentation of graph-based RAG methods by proposing a four-stage unified framework (graph building, index construction, operator configuration, retrieval & generation) and re-implementing 12 representative approaches to compare them across 11 QA datasets. It introduces new variants such as VGraphRAG and CheapRAG, and provides an open-source testbed to enable reproducible, fine-grained evaluation of retrieval components and costs. Key findings show that graph-based RAG generally improves over vanilla RAG, with high-level community summaries helping abstract QA and vector-based retrieval benefiting complex reasoning, albeit at higher computational costs. The work offers practical guidelines and opportunities for future research, including dynamic knowledge sources, graph quality assessment, and integration with heterogeneous data sources and graph DBMS.

Abstract

Graph-based Retrieval-Augmented Generation (RAG) has proven effective in integrating external knowledge into large language models (LLMs), improving their factual accuracy, adaptability, interpretability, and trustworthiness. A number of graph-based RAG methods have been proposed in the literature. However, these methods have not been systematically and comprehensively compared under the same experimental settings. In this paper, we first summarize a unified framework to incorporate all graph-based RAG methods from a high-level perspective. We then extensively compare representative graph-based RAG methods over a range of questing-answering (QA) datasets -- from specific questions to abstract questions -- and examine the effectiveness of all methods, providing a thorough analysis of graph-based RAG approaches. As a byproduct of our experimental analysis, we are also able to identify new variants of the graph-based RAG methods over specific QA and abstract QA tasks respectively, by combining existing techniques, which outperform the state-of-the-art methods. Finally, based on these findings, we offer promising research opportunities. We believe that a deeper understanding of the behavior of existing methods can provide new valuable insights for future research.

Figures (19)

• Figure 1: Overview of vanilla RAG and graph-based RAG.
• Figure 2: Workflow of graph-based RAG methods under our unified framework.
• Figure 3: Workflow of our empirical study.
• Figure 4: Token cost of graph building on specific QA datasets.
• Figure 5: Token cost of index construction in specific QA.