A Multi-Granularity Retrieval Framework for Visually-Rich Documents

TL;DR

The paper addresses multimodal retrieval for visually-rich documents by extending retrieval beyond text to include images, tables, and charts. It introduces a training-free, multi-granularity framework that unifies two benchmarks, MMDocIR and M2KR, using hierarchical encodings, modality-aware retrieval signals, and vision-language model (VLM) based candidate verification. The framework comprises two tracks (M2KR and MMDocIR) with dedicated instantiations, coupled by a fusion mechanism (including Reciprocal Rank Fusion) and a VLM verification step, achieving robust performance without task-specific fine-tuning. The approach demonstrates significant gains from layout-aware search and cross-modal verification, with a top score of 65.56 on the evaluated tasks, underscoring scalability and reproducibility for real-world multimodal document retrieval.

Abstract

Retrieval-augmented generation (RAG) systems have predominantly focused on text-based retrieval, limiting their effectiveness in handling visually-rich documents that encompass text, images, tables, and charts. To bridge this gap, we propose a unified multi-granularity multimodal retrieval framework tailored for two benchmark tasks: MMDocIR and M2KR. Our approach integrates hierarchical encoding strategies, modality-aware retrieval mechanisms, and vision-language model (VLM)-based candidate filtering to effectively capture and utilize the complex interdependencies between textual and visual modalities. By leveraging off-the-shelf vision-language models and implementing a training-free hybrid retrieval strategy, our framework demonstrates robust performance without the need for task-specific fine-tuning. Experimental evaluations reveal that incorporating layout-aware search and VLM-based candidate verification significantly enhances retrieval accuracy, achieving a top performance score of 65.56. This work underscores the potential of scalable and reproducible solutions in advancing multimodal document retrieval systems.

Figures (3)

• Figure 1: Overview of the Multi-Granularity Multimodal Retrieval Framework for M2KR.
• Figure 2: Overview of the MMDocIR retrieval framework combining full-page and regional information.
• Figure 3: Performance trend across progressive improvements.