M$^3$KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation

TL;DR

This work addresses the challenges of multimodal retrieval for audio-visual tasks by introducing $M^3KG$-RAG, a multi-hop multimodal knowledge graph-enhanced retrieval-augmented generation framework. It combines a lightweight, multi-agent pipeline to construct a context-rich MMKG with three steps—context-enriched triplet extraction, knowledge grounding, and context-aware refinement—plus a Self-Reflection Loop to ensure quality. The retrieval stage employs modality-wise search and GRASP to ground and prune evidence before graph-augmented generation conditions the MLLM on relevant triplets and refined descriptions. Across Audio QA, Video QA, and AV QA benchmarks, $M^3KG$-RAG yields substantial improvements over baselines, and its benefits amplify with stronger LLMs like GPT-4o, demonstrating improved reasoning depth and grounding fidelity in multimodal contexts.

Abstract

Retrieval-Augmented Generation (RAG) has recently been extended to multimodal settings, connecting multimodal large language models (MLLMs) with vast corpora of external knowledge such as multimodal knowledge graphs (MMKGs). Despite their recent success, multimodal RAG in the audio-visual domain remains challenging due to 1) limited modality coverage and multi-hop connectivity of existing MMKGs, and 2) retrieval based solely on similarity in a shared multimodal embedding space, which fails to filter out off-topic or redundant knowledge. To address these limitations, we propose M$^3$KG-RAG, a Multi-hop Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving reasoning depth and answer faithfulness in MLLMs. Specifically, we devise a lightweight multi-agent pipeline to construct multi-hop MMKG (M$^3$KG), which contains context-enriched triplets of multimodal entities, enabling modality-wise retrieval based on input queries. Furthermore, we introduce GRASP (Grounded Retrieval And Selective Pruning), which ensures precise entity grounding to the query, evaluates answer-supporting relevance, and prunes redundant context to retain only knowledge essential for response generation. Extensive experiments across diverse multimodal benchmarks demonstrate that M$^3$KG-RAG significantly enhances MLLMs' multimodal reasoning and grounding over existing approaches.

Figures (8)

• Figure 1: Illustration of multimodal RAG scenarios. Incorrect answers are shown in red, correct answers in blue. (a) Shared embedding search misaligns with the audio-visual query. (b) Noisy, single-hop facts provide little answer support. (c) M$^3$KG-RAG uses modality-wise multi-hop retrieval for answer-supporting context.
• Figure 2: An overview of the M$^3$KG construction pipeline. The pipeline consists of three steps: (i) Context-Enriched Triplet Extraction, which rewrites multimodal captions into knowledge-intensive text and extracts entity–relation triplets; (ii) Knowledge Grounding, linking normalized entities to open knowledge bases to obtain candidate descriptions; (iii) Context-Aware Description Refinement, selecting and rewriting the most context-relevant descriptions for each entity; and Self-Reflection Loop, where an inspector agent validates or re-runs uncertain outputs to ensure graph quality.
• Figure 3: Overview of the Multimodal RAG framework. The framework consists of two components: (a) Modality-Wise Retrieval, which retrieves multi-hop triplets aligned with the query from the M$^3$KG; and (b) GRASP (Grounded Retrieval And Selective Pruning), which uses visual and/or audio grounding models to check entity presence and prunes triplets that are off-topic or non-informative. The resulting subgraph is then provided to an MLLM for query-relevant, evidence-grounded audio-visual reasoning.
• Figure 4: Qualitative results on various Question Answering tasks. Incorrect and insufficient model responses are highlighted in red, while correct and sufficient responses are highlighted in blue.
• Figure 5: Sensitivity analysis. M.J. score on VALOR versus modality-wise distance threshold $\tau$ (top) and GRASP presence threshold $\eta_{av}$ (bottom).