M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

TL;DR

M3DocRAG introduces a unified multi-modal retrieval-and-generation framework that extends DocVQA to open-domain, multi-page, and multi-document settings. It embeds page visuals, retrieves relevant pages with a multimodal retriever, and answers via a multi-modal language model, enabling efficient processing of thousands of pages across documents. The authors also present M3DocVQA, a challenging open-domain benchmark with 2,441 questions over 3,368 PDFs (41,005 pages) to evaluate retrieval and cross-modal reasoning. Across open- and closed-domain benchmarks, M3DocRAG achieves state-of-the-art results, demonstrates strong handling of non-text evidence, and provides extensive ablations on indexing, LMs, and retrievers. This work advances practical document understanding by integrating scalable multi-modal retrieval with generation for large-scale, real-world document corpora.

Abstract

Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.

Figures (7)

• Figure 1: Comparison of multi-modal document understanding pipelines. Previous works focus on (a) Single-page DocVQA that cannot handle many long documents or (b) Text-based RAG that ignores visual information. Our (c) M3DocRAG framework retrieves relevant documents and answers questions using multi-modal retrieval and MLM components, so that it can efficiently handle many long documents while preserving visual information.
• Figure 2: Comparison of existing DocVQA datasets (left; e.g., DocVQA mathew2021docvqa) and our M3DocVQA dataset (right). In contrast to previous DocVQA datasets that have questions that are specific to a single provided PDF (e.g., "What was the gross profit in the year 2009?"), M3DocVQA has information-seeking questions that benchmark open-domain question answering capabilities across more than 3,000 PDF documents (i.e., 40,000+ pages).
• Figure 3: Our M3DocRAG framework (\ref{['sec:method']}) consists of three stages: (1) document embedding (\ref{['sec:embedding']}), (2) page retrieval (\ref{['sec:retrieval']}), and (3) question answering (\ref{['sec:question_answering']}). In (1) document embedding, we extract visual embedding (with ColPali) to represent each page from all PDF documents. In (2) page retrieval, we retrieve the top-K pages of high relevance (MaxSim scores) with text queries. In an open-domain setting, we create approximate page indices for faster search. In (3) question answering, we conduct visual question answering with multi-modal LM (e.g. Qwen2-VL) to obtain the final answer.
• Figure 4: Illustration of PDF collections in M3DocVQA. We first collect the URLs of all supporting contexts (Wikipedia documents) of individual questions of MultimodalQA talmor2021multimodalqa. Then, we create PDF versions from their URLs by rendering them in a web browser.
• Figure 5: Qualitative example of ColPali + Qwen2-VL 7B on M3DocVQA. Image regions relevant to the question/answer are highlighted with orange boxes. The answer information is only stored visually within the game logo, where a man is leaning on a motorcycle.