UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG

TL;DR

UniDoc-Bench presents the first large-scale, document-centric multimodal RAG benchmark, built from 70k real-world PDF pages across 8 domains to enable fair comparisons across text-only, image-only, text-image fusion, and multimodal retrieval pipelines. Through a 1,600 QA-pair dataset grounded in text, figures, and tables, the authors provide a standardized evaluation protocol, fixed candidate pools, and cross-modality grounding to fairly assess retrieval and end-to-end generation. Experiments reveal that text-image fusion RAG consistently outperforms unimodal and joint multimodal embeddings, while current multimodal embeddings lag behind fusion of separate text and image retrievers. The benchmark also uncovers when and how visual context complements textual evidence, identifies pervasive challenges for image-dependent queries, and offers concrete guidance for building more robust, faithful MM-RAG pipelines with broad real-world impact.

Abstract

Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models (LLMs) and agents to real-world knowledge bases, yet current evaluations are fragmented, focusing on either text or images in isolation or on simplified multimodal setups that fail to capture document-centric multimodal use cases. In this paper, we introduce UniDoc-Bench, the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages across eight domains. Our pipeline extracts and links evidence from text, tables, and figures, then generates 1,600 multimodal QA pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries. To ensure reliability, 20% of QA pairs are validated by multiple annotators and expert adjudication. UniDoc-Bench supports apples-to-apples comparison across four paradigms: (1) text-only, (2) image-only, (3) multimodal text-image fusion, and (4) multimodal joint retrieval -- under a unified protocol with standardized candidate pools, prompts, and evaluation metrics. Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval, indicating that neither text nor images alone are sufficient and that current multimodal embeddings remain inadequate. Beyond benchmarking, our analysis reveals when and how visual context complements textual evidence, uncovers systematic failure modes, and offers actionable guidance for developing more robust MM-RAG pipelines.

Figures (8)

• Figure 1: UniDoc-Bench overview.
• Figure 2: Data Construction pipeline. (a) We filter and tag PDFA documents to curate a high-quality database of $70$k pages spanning $8$ domains. (b) We parse documents into text, figures, and tables, then synthesize initial QA pairs covering four question types and three modalities using adapted templates. (c) We ground answers in supporting evidence, refine questions for human-intent and self-containment, and verify responses for factuality and completeness, yielding $1,600$ QA pairs. To ensure quality, $20\%$ of the dataset is validated by three independent human annotators.
• Figure 3: Example of PDF parsing with figure placeholders (<<fig-XXX>>).
• Figure 4: Image-retrieval system fails to extract factual facts and details.
• Figure 5: Image-retrieval system fails to extract factual facts and details in the image.