Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding

TL;DR

This survey addresses the challenge of understanding visually rich documents by integrating multimodal retrieval-augmented generation (RAG). It proposes a taxonomy across domain openness, retrieval modality, and retrieval granularity, and examines graph-based and agent-based hybrid architectures that enhance grounding and reasoning. The work consolidates datasets, benchmarks, and real-world applications in finance, science, and social science, while highlighting efficiency, fine-grained representation, and robustness as central open issues. Together, these insights provide a roadmap for scalable, reliable multimodal document AI that can operate across large corpora and complex layouts.

Abstract

Document understanding is critical for applications from financial analysis to scientific discovery. Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs), face key limitations: the former loses structural detail, while the latter struggles with context modeling. Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, i.e., combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG. This approach enables holistic retrieval and reasoning across all modalities, unlocking comprehensive document intelligence. Recognizing its importance, this paper presents a systematic survey of Multimodal RAG for document understanding. We propose a taxonomy based on domain, retrieval modality, and granularity, and review advances involving graph structures and agentic frameworks. We also summarize key datasets, benchmarks, and applications, and highlight open challenges in efficiency, fine-grained representation, and robustness, providing a roadmap for future progress in document AI.

Figures (5)

• Figure 1: Impact and research progress of Multimodal RAG for document understanding: (a) MLLMs with and without Multimodal RAG for large document comprehension. (b) Growth in related publications from 2024 to 2025.
• Figure 2: Comparison between closed-domain and open-domain multimodal RAG. (a) In the closed domain, the model leverages in-document retrieval from a single document to answer context-specific questions. (b) In the open domain, the model relies on cross-document retrieval from multiple documents to answer open-ended questions.
• Figure 3: Comparison of retrieval modality: (a) image-based RAG retrieves information solely from page images, offering efficiency but limited textual detail; (b) image+text-based RAG integrates OCR/annotations with visual features, enabling richer retrieval at the cost of higher processing complexity.
• Figure 4: Comparison of retrieval granularity in multimodal document search. (a) Page-level: entire pages are encoded and ranked as whole units. (b) Element-level: pages are decomposed into tables, charts, images, and text blocks; retrieval operates over these elements to localize evidence and aggregate results.
• Figure 5: Hybrid enhancements for multimodal RAG. (a) Graph-based: documents/elements form a graph index, and retrieval proceeds via graph traversal to surface relevant neighborhoods. (b) Agent-based: an LLM agent decomposes the text query, orchestrates multimodal retrieval, verifies the gathered evidence, and synthesizes the final answer.