Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval

Rashmi R, Vidyadhar Upadhya

TL;DR

This work tackles the challenge of retrieval-augmented generation over unstructured multimodal data by proposing MAHA, a Modality-Aware Hybrid Retrieval Architecture that fuses dense vector retrieval with a modality-aware knowledge graph to enable cross-modal, multi-hop reasoning. MAHA constructs a multimodal KG with nodes representing text, images, tables, equations, and graphs, and edges that capture cross-modal relationships, enabling structured reasoning alongside semantic similarity. Evaluations on diverse multimodal benchmarks show MAHA achieving a ROUGE-L of 0.486, Recall@3 of 0.81, and MRR of 0.74, with full modality coverage, outperforming baselines and existing multimodal RAG frameworks. The results demonstrate a scalable, explainable retrieval framework that advances RAG systems by effectively leveraging cross-modal evidence for complex document understanding and generation.

Abstract

Current Retrieval-Augmented Generation (RAG) systems primarily operate on unimodal textual data, limiting their effectiveness on unstructured multimodal documents. Such documents often combine text, images, tables, equations, and graphs, each contributing unique information. In this work, we present a Modality-Aware Hybrid retrieval Architecture (MAHA), designed specifically for multimodal question answering with reasoning through a modality-aware knowledge graph. MAHA integrates dense vector retrieval with structured graph traversal, where the knowledge graph encodes cross-modal semantics and relationships. This design enables both semantically rich and context-aware retrieval across diverse modalities. Evaluations on multiple benchmark datasets demonstrate that MAHA substantially outperforms baseline methods, achieving a ROUGE-L score of 0.486, providing complete modality coverage. These results highlight MAHA's ability to combine embeddings with explicit document structure, enabling effective multimodal retrieval. Our work establishes a scalable and interpretable retrieval framework that advances RAG systems by enabling modality-aware reasoning over unstructured multimodal data.

Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval

TL;DR

This work tackles the challenge of retrieval-augmented generation over unstructured multimodal data by proposing MAHA, a Modality-Aware Hybrid Retrieval Architecture that fuses dense vector retrieval with a modality-aware knowledge graph to enable cross-modal, multi-hop reasoning. MAHA constructs a multimodal KG with nodes representing text, images, tables, equations, and graphs, and edges that capture cross-modal relationships, enabling structured reasoning alongside semantic similarity. Evaluations on diverse multimodal benchmarks show MAHA achieving a ROUGE-L of 0.486, Recall@3 of 0.81, and MRR of 0.74, with full modality coverage, outperforming baselines and existing multimodal RAG frameworks. The results demonstrate a scalable, explainable retrieval framework that advances RAG systems by effectively leveraging cross-modal evidence for complex document understanding and generation.

Abstract

Current Retrieval-Augmented Generation (RAG) systems primarily operate on unimodal textual data, limiting their effectiveness on unstructured multimodal documents. Such documents often combine text, images, tables, equations, and graphs, each contributing unique information. In this work, we present a Modality-Aware Hybrid retrieval Architecture (MAHA), designed specifically for multimodal question answering with reasoning through a modality-aware knowledge graph. MAHA integrates dense vector retrieval with structured graph traversal, where the knowledge graph encodes cross-modal semantics and relationships. This design enables both semantically rich and context-aware retrieval across diverse modalities. Evaluations on multiple benchmark datasets demonstrate that MAHA substantially outperforms baseline methods, achieving a ROUGE-L score of 0.486, providing complete modality coverage. These results highlight MAHA's ability to combine embeddings with explicit document structure, enabling effective multimodal retrieval. Our work establishes a scalable and interpretable retrieval framework that advances RAG systems by enabling modality-aware reasoning over unstructured multimodal data.
Paper Structure (18 sections, 5 figures)

This paper contains 18 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Prior Work, Limitations and Our Contributions
  3. Modality-Aware Hybrid retrieval Architecture (MAHA)
  4. Architecture
  5. Ingestion and Embedding:
  6. Vectorstore Indexing and Knowledge Graph Construction:
  7. Hybrid Retrieval (Vector + Graph):
  8. Context-Aware Generation with LLMs:
  9. Experimental Setup
  10. Datasets
  11. Baseline Retrieval Systems
  12. Evaluation Metrics
  13. Challenges
  14. Experimental Results
  15. MAHA vs Baseline Retrieval Systems
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The proposed Modality-Aware Hybrid retrieval Architecture (MAHA).
  • Figure 2: Text and Table, Text and Image Relationships captured through KGs
  • Figure 3: Performance of MAHA compared with baseline retrievers.
  • Figure 4: Performance and Modality Coverage of MAHA compared with KG-based multimodal RAG frameworks.
  • Figure 5: Performance of MAHA compared with vector-only and graph-only retrievers.