Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generation

TL;DR

ParallaxRAG addresses hallucination and fragile reasoning in KG-grounded QA by introducing a symmetric, multi-view KG-RAG framework that decouples queries and graph triples into head-specific representations. The approach leverages Pairwise Similarity Regulation to maintain head diversity and a query-aware gating mechanism to emphasize the most relevant heads, enabling grounded, step-wise multi-hop reasoning. Empirical results on WebQSP, CWQ, and BioASQ show strong retrieval quality, interpretable head specialization, and improved end-to-end QA with reduced hallucinations, achieving competitive or state-of-the-art performance. This work provides a principled direction for robust, knowledge-grounded multi-hop reasoning and offers practical benefits for scalable, domain-general KGQA systems.

Abstract

Large language models (LLMs) excel at language understanding but often hallucinate and struggle with multi-hop reasoning. Knowledge-graph-based retrieval-augmented generation (KG-RAG) offers grounding, yet most methods rely on flat embeddings and noisy path exploration. We propose ParallaxRAG, a framework that symmetrically decouples queries and graph triples into multi-view spaces, enabling a robust retrieval architecture that explicitly enforces head diversity while constraining weakly related paths. Central to our approach is the observation that different attention heads specialize in semantic relations at distinct reasoning stages, contributing to different hops of the reasoning chain. This specialization allows ParallaxRAG to construct cleaner subgraphs and guide LLMs through grounded, step-wise reasoning. Experiments on WebQSP and CWQ, under our unified, reproducible setup (BGE-M3 + Llama3.1-8B), demonstrate competitive retrieval and QA performance, alongside reduced hallucination and good generalization. Our results highlight multi-view head specialization as a principled direction for knowledge-grounded multi-hop reasoning. Our implementation will be released as soon as the paper is accepted.

Figures (9)

• Figure 1: Comparison of standard KG-RAG and multi-head KG-RAG embeddings. In ParallaxRAG, attention heads specialize in distinct semantic relations across reasoning stages, contributing to successive hops in the reasoning chain.
• Figure 2: The architecture of ParallaxRAG, illustrating its three main stages: Preprocess, Retrieval, and Generate. The core retrieval process begins by creating specialized multi-view representations for both the graph (b) and the question (c), a process guided by directional distance encoding and regularized by pairwise similarity (a). Next, candidate triples are scored in parallel across these diverse views (d). Finally, a query-aware gating mechanism computes head importance (e) to produce a weighted aggregation of the scores, identifying the most relevant triples for the final generation stage.
• Figure 3: Visualization of attention head specialization, revealing a task-adaptive division of labor. The heatmaps display three metrics (Contribution, Use Rate, Hit Rate) for the top 5 specialist heads, selected from our BGE-based model's 16 heads according to their overall contribution. The Others row aggregates the remaining 11 heads: for Contribution, this value is the sum of their contributions, while for Use Rate and Hit Rate, it is their average, see more explanation in (Appendix \ref{['app:head']}). We define steps as the BFS expansion depth along reasoning paths, rather than the conventional hop (shortest-path length), in order to capture head behaviors at each layer of candidate exploration. (a) On WebQSP (short chains). (b) On CWQ (long chains).
• Figure 4: Sensitivity analysis of the PSR strength parameter $\beta$ on the generalization task, where the retriever was trained on WebQSP and tested on CWQ, using Llama-3.1-8B as the generator.
• Figure 5: Attention Head Specialization Analysis on CWQ Dataset with Alternative Backbone Encoders. (a) Results using the E5-large-v2 model. (b) Results using the GTE-large model.