Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey

Xiantao Zhang

TL;DR

This paper tackles the challenge of retrieval-augmented generation over visually rich documents (VRDs), where meaning arises from text, layout, and graphics, and OCR-based pipelines introduce errors. It proposes a three-role taxonomy for Multimodal Large Language Models (MLLMs) in VRD RAG: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers, and analyzes their trade-offs in retrieval granularity, information fidelity, latency, and index size. The authors review methods, practical deployments, and empirical evidence across these roles, offering guidance on when each approach is advantageous and highlighting open challenges such as adaptive retrieval, efficiency, and evaluation metrics. The work aims to guide practitioners in building reliable, cost-aware VRD RAG systems and to steer future research toward adaptive, scalable, and well-evaluated multimodal retrieval for document understanding.

Abstract

Visually rich documents (VRDs) challenge retrieval-augmented generation (RAG) with layout-dependent semantics, brittle OCR, and evidence spread across complex figures and structured tables. This survey examines how Multimodal Large Language Models (MLLMs) are being used to make VRD retrieval practical for RAG. We organize the literature into three roles: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers. We compare these roles along retrieval granularity, information fidelity, latency and index size, and compatibility with reranking and grounding. We also outline key trade-offs and offer some practical guidance on when to favor each role. Finally, we identify promising directions for future research, including adaptive retrieval units, model size reduction, and the development of evaluation methods.

Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey

TL;DR

This paper tackles the challenge of retrieval-augmented generation over visually rich documents (VRDs), where meaning arises from text, layout, and graphics, and OCR-based pipelines introduce errors. It proposes a three-role taxonomy for Multimodal Large Language Models (MLLMs) in VRD RAG: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers, and analyzes their trade-offs in retrieval granularity, information fidelity, latency, and index size. The authors review methods, practical deployments, and empirical evidence across these roles, offering guidance on when each approach is advantageous and highlighting open challenges such as adaptive retrieval, efficiency, and evaluation metrics. The work aims to guide practitioners in building reliable, cost-aware VRD RAG systems and to steer future research toward adaptive, scalable, and well-evaluated multimodal retrieval for document understanding.

Abstract

Visually rich documents (VRDs) challenge retrieval-augmented generation (RAG) with layout-dependent semantics, brittle OCR, and evidence spread across complex figures and structured tables. This survey examines how Multimodal Large Language Models (MLLMs) are being used to make VRD retrieval practical for RAG. We organize the literature into three roles: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers. We compare these roles along retrieval granularity, information fidelity, latency and index size, and compatibility with reranking and grounding. We also outline key trade-offs and offer some practical guidance on when to favor each role. Finally, we identify promising directions for future research, including adaptive retrieval units, model size reduction, and the development of evaluation methods.
Paper Structure (43 sections, 2 figures, 3 tables)

This paper contains 43 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Scope and goal
  3. Contributions
  4. Organization
  5. Related Work
  6. Retrieval-Augmented Generation
  7. Multimodal Retrieval
  8. Multimodal Large Language Models
  9. Related Surveys
  10. Three Roles of MLLMs in VRD RAG
  11. MLLMs as Modality-Unifying Captioners
  12. From captioning to document textualization
  13. Captions as textual proxies
  14. Practical Deployments
  15. Video-RAG as a mirror for VRDs
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Overview of how MLLMs enter VRD retrieval for RAG across three roles. Left:Modality-Unifying Captioners (§\ref{['subsec:mllm-as-captioner']}); Middle:Multimodal Embedders (§\ref{['subsec:mllm-as-mmemb']}); Right:End-to-End Representers (§\ref{['subsec:mllm-as-e2erepr']}). Each panel sketches the pipeline from document intake to retrieval and answer synthesis, highlighting typical retrieval units and index types.
  • Figure 2: Comparison of encoding latency (displaying top 1%, 50%, and 99th percentiles) and vector search latency for the CLIPSFwei2024uniir and UniEmb lin2025mmembed models. Measurements were based on 100 randomly sampled queries from each of the 16 M-BEIR wei2024uniir tasks.