Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

TL;DR

This work introduces Amar, an adaptive multi-aspect retrieval-augmented KGQA framework that leverages entities, relations, and subgraphs from knowledge graphs as prompt embeddings. It deploys two novel modules—self-alignment to fuse cross-aspect information and relevance gating to softly filter retrieved data—along with a query-refinement stage to produce executable S-expressions converted to SPARQL for KG answers. Empirical results on WebQSP and CWQ demonstrate state-of-the-art accuracy and substantial improvements in logical-form generation compared to direct-context baselines and 22 strong competitors. The approach reduces noise from long retrievals, enables efficient fine-tuning via LoRA, and shows robust performance across LLM backbones, signaling practical impact for reliable KG reasoning in real-world QA systems.

Abstract

Large Language Models (LLMs) demonstrate remarkable capabilities, yet struggle with hallucination and outdated knowledge when tasked with complex knowledge reasoning, resulting in factually incorrect outputs. Previous studies have attempted to mitigate it by retrieving factual knowledge from large-scale knowledge graphs (KGs) to assist LLMs in logical reasoning and prediction of answers. However, this kind of approach often introduces noise and irrelevant data, especially in situations with extensive context from multiple knowledge aspects. In this way, LLM attention can be potentially mislead from question and relevant information. In our study, we introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. The Amar framework comprises two key sub-components: 1) a self-alignment module that aligns commonalities among entities, relations, and subgraphs to enhance retrieved text, thereby reducing noise interference; 2) a relevance gating module that employs a soft gate to learn the relevance score between question and multi-aspect retrieved data, to determine which information should be used to enhance LLMs' output, or even filtered altogether. Our method has achieved state-of-the-art performance on two common datasets, WebQSP and CWQ, showing a 1.9\% improvement in accuracy over its best competitor and a 6.6\% improvement in logical form generation over a method that directly uses retrieved text as context prompts. These results demonstrate the effectiveness of Amar in improving the reasoning of LLMs.

Figures (4)

• Figure 1: An example illustrates the retrieval results categorized as relevant, irrelevant, and partially relevant.
• Figure 2: Overall framework of Amar . We retrieve multi-aspect knowledge from KG and obtain weighted consistency prompt embeddings using Self-Alignment and Relevance Gating modules. These embeddings are combined with the question and fed into LLMs. The generated logical expression is refined and used to query KG, ultimately getting final answers. The unlabeled color blocks in the middle represent the tokens input to the LLM, with pink blocks denoting question tokens and blue blocks denoting retrieved knowledge tokens.
• Figure 3: Performance vary with the number of Retrieval.
• Figure 4: Combine with different fine-tuning methods.