Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment

TL;DR

This work tackles hallucinations in GraphRAG by probing how LLMs attend to and ground linearized graph inputs. It introduces two mechanistic metrics, PRD and SAS, to diagnose attention concentration on shortest paths and semantic grounding in retrieved knowledge, respectively, and shows that hallucinations correlate with high PRD and low SAS. Building on these signals, the authors train a lightweight detector (GGA) using PRD, SAS, and simple surface features, achieving state-of-the-art AUC and F1 on two LLMs without model fine-tuning. The study provides actionable insights into grounding graph-structured knowledge in LLMs and offers a practical tool for improving GraphRAG reliability, with clear directions for extending to multi-hop and diverse model architectures.

Abstract

Graph-based Retrieval-Augmented Generation (GraphRAG) enhances Large Language Models (LLMs) by incorporating external knowledge from linearized subgraphs retrieved from knowledge graphs. However, LLMs struggle to interpret the relational and topological information in these inputs, resulting in hallucinations that are inconsistent with the retrieved knowledge. To analyze how LLMs attend to and retain structured knowledge during generation, we propose two lightweight interpretability metrics: Path Reliance Degree (PRD), which measures over-reliance on shortest-path triples, and Semantic Alignment Score (SAS), which assesses how well the model's internal representations align with the retrieved knowledge. Through empirical analysis on a knowledge-based QA task, we identify failure patterns associated with over-reliance on salient paths and weak semantic grounding, as indicated by high PRD and low SAS scores. We further develop a lightweight post-hoc hallucination detector, Graph Grounding and Alignment (GGA), which outperforms strong semantic and confidence-based baselines across AUC and F1. By grounding hallucination analysis in mechanistic interpretability, our work offers insights into how structural limitations in LLMs contribute to hallucinations, informing the design of more reliable GraphRAG systems in the future.

Figures (7)

• Figure 1: Hallucinations in GraphRAG during Knowledge Base Question Answering (KBQA).
• Figure 2: The model over-focuses on shortest paths in the knowledge graph, neglecting relevant context triples (Challenge 1), and the feed-forward layers drift away from retrieved knowledge, relying instead on parametric memory (Challenge 2), leading to hallucinated outputs even when accurate subgraph information is available.
• Figure 3: Prompt template for GraphRAG question answering showing linearized knowledge graph triples and answer format requirements.
• Figure 4: Box plots of PRD and SAS scores for hallucinated vs. truthful responses. Hallucinated answers show higher PRD and lower SAS, indicating stronger shortest path reliance and weaker semantic grounding.
• Figure 5: Feature space and correlation. (Left) Scatter plot of TUS vs. SAS shows overlapping but distinguishable clusters. (Right) Correlation matrix shows weak dependencies between features and label, suggesting complementary information.