MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation

TL;DR

MegaRAG addresses the challenge of long-form multimodal document understanding under constrained context windows by constructing a Multimodal Knowledge Graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ from per-page inputs and refining it with cross-page context. It combines lightweight page-specific subgraphs with a global KG, embedding entities, relations, and pages in a shared multimodal space for retrieval, and employs a two-stage MMKG-augmented generation to fuse textual and visual evidence. Extensive experiments on textual and multimodal datasets show MegaRAG outperforms GraphRAG and LightRAG on both global and local QA, with ablations confirming the critical role of MMKG-based retrieval and cross-modal refinement. This approach offers a scalable, interpretable pathway to cross-modal reasoning in RAG systems, with practical impact for complex document understanding and knowledge extraction across domains.

Abstract

Retrieval-augmented generation (RAG) enables large language models (LLMs) to dynamically access external information, which is powerful for answering questions over previously unseen documents. Nonetheless, they struggle with high-level conceptual understanding and holistic comprehension due to limited context windows, which constrain their ability to perform deep reasoning over long-form, domain-specific content such as full-length books. To solve this problem, knowledge graphs (KGs) have been leveraged to provide entity-centric structure and hierarchical summaries, offering more structured support for reasoning. However, existing KG-based RAG solutions remain restricted to text-only inputs and fail to leverage the complementary insights provided by other modalities such as vision. On the other hand, reasoning from visual documents requires textual, visual, and spatial cues into structured, hierarchical concepts. To address this issue, we introduce a multimodal knowledge graph-based RAG that enables cross-modal reasoning for better content understanding. Our method incorporates visual cues into the construction of knowledge graphs, the retrieval phase, and the answer generation process. Experimental results across both global and fine-grained question answering tasks show that our approach consistently outperforms existing RAG-based approaches on both textual and multimodal corpora.

Figures (8)

• Figure 1: Overview of our MegaRAG for MMKG construction and MMKG-augmented generation. (a) Initial Construction: Multimodal inputs from each page are processed by an MLLM to extract entities and relations $(E,R)^0_i$ in parallel. The page-level results are then joined by aligning identical entity names and relations, forming the initial document-level MMKG $\mathcal{G}^0$. (b) Refinement: Each page retrieves a subgraph $\mathcal{G}^{0}_{i}$ from $\mathcal{G}^0$ to assist the MLLM in refining the initial graph, yielding $\mathcal{G}^1$. (c) Indexing: The refined MMKG is encoded by an MMRAG's retrieval approach into dense entity, relation, and page embeddings for efficient retrieval. (d) Retrieval & Answer Generation: A user query is parsed into low- and high-level keywords for retrieving relevant subgraphs and pages. These are fed into the MLLM for 2-stage answer generation.
• Figure 2: Prompt for extracting entities and relations during the initial construction of the MMKG.
• Figure 3: Prompt for MMKG refinement stage.
• Figure 4: Prompts for MMKG-augmented answer generation. (a) Generates an intermediate answer from the retrieved pages. (b) Generates an intermediate answer from the retrieved MMKG subgraph. (c) The final answer is produced by combining both intermediate responses.
• Figure 5: (a) Prompt used for global question generation. (b) Example global questions.