Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering GraphRAG with Knowledge Filtering and Integration

Kai Guo, Harry Shomer, Shenglai Zeng, Haoyu Han, Yu Wang, Jiliang Tang

TL;DR

This work tackles hallucination and knowledge gaps in graph retrieval-augmented generation for KGQA. It introduces GraphRAG-FI, combining a two-stage filtering module with a logits-based integration strategy to suppress noisy retrieval and to balance external knowledge with the LLM’s intrinsic reasoning. Empirical results on WebQSP and CWQ across multiple backbones show consistent improvements over baselines, with robust performance under noise and clear ablations demonstrating the contributions of filtering and integration. The approach yields a more reliable GraphRAG framework with practical impact for multi-hop knowledge graph reasoning.

Abstract

In recent years, large language models (LLMs) have revolutionized the field of natural language processing. However, they often suffer from knowledge gaps and hallucinations. Graph retrieval-augmented generation (GraphRAG) enhances LLM reasoning by integrating structured knowledge from external graphs. However, we identify two key challenges that plague GraphRAG:(1) Retrieving noisy and irrelevant information can degrade performance and (2)Excessive reliance on external knowledge suppresses the model's intrinsic reasoning. To address these issues, we propose GraphRAG-FI (Filtering and Integration), consisting of GraphRAG-Filtering and GraphRAG-Integration. GraphRAG-Filtering employs a two-stage filtering mechanism to refine retrieved information. GraphRAG-Integration employs a logits-based selection strategy to balance external knowledge from GraphRAG with the LLM's intrinsic reasoning,reducing over-reliance on retrievals. Experiments on knowledge graph QA tasks demonstrate that GraphRAG-FI significantly improves reasoning performance across multiple backbone models, establishing a more reliable and effective GraphRAG framework.

Empowering GraphRAG with Knowledge Filtering and Integration

TL;DR

This work tackles hallucination and knowledge gaps in graph retrieval-augmented generation for KGQA. It introduces GraphRAG-FI, combining a two-stage filtering module with a logits-based integration strategy to suppress noisy retrieval and to balance external knowledge with the LLM’s intrinsic reasoning. Empirical results on WebQSP and CWQ across multiple backbones show consistent improvements over baselines, with robust performance under noise and clear ablations demonstrating the contributions of filtering and integration. The approach yields a more reliable GraphRAG framework with practical impact for multi-hop knowledge graph reasoning.

Abstract

In recent years, large language models (LLMs) have revolutionized the field of natural language processing. However, they often suffer from knowledge gaps and hallucinations. Graph retrieval-augmented generation (GraphRAG) enhances LLM reasoning by integrating structured knowledge from external graphs. However, we identify two key challenges that plague GraphRAG:(1) Retrieving noisy and irrelevant information can degrade performance and (2)Excessive reliance on external knowledge suppresses the model's intrinsic reasoning. To address these issues, we propose GraphRAG-FI (Filtering and Integration), consisting of GraphRAG-Filtering and GraphRAG-Integration. GraphRAG-Filtering employs a two-stage filtering mechanism to refine retrieved information. GraphRAG-Integration employs a logits-based selection strategy to balance external knowledge from GraphRAG with the LLM's intrinsic reasoning,reducing over-reliance on retrievals. Experiments on knowledge graph QA tasks demonstrate that GraphRAG-FI significantly improves reasoning performance across multiple backbone models, establishing a more reliable and effective GraphRAG framework.

Paper Structure

This paper contains 27 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary studies
  4. Experimental settings
  5. The Impact of GraphRAG
  6. Positive Impact of GraphRAG
  7. Limited Impact of GraphRAG
  8. Negative Impact of GraphRAG
  9. The Impact of the Number of Retrieved Paths
  10. Attention Reflects the Importance of Retrieved Information
  11. Internal Knowledge Filtering
  12. Discussions
  13. Method
  14. GraphRAG-Filtering
  15. Stage 1: Coarse Filtering using Attention:
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Category A includes cases where both GraphRAG and the LLM-only model are correct. Category B covers instances where GraphRAG outperforms the LLM-only model, while Category C includes cases where the LLM-only model performs better than GraphRAG. Category D represents cases where both models fail.
  • Figure 2: The relationship between path number and average F1
  • Figure 3: Attention Scores for Retrieved Information With/Without Ground Truth
  • Figure 4: An overview of the GraphRAG-FI framework.
  • Figure 5: An Example of Our Prompt