URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding

TL;DR

URaG addresses efficient long-document understanding in multimodal LLMs by unifying retrieval and generation within a single model. It exploits a lightweight cross-modal retrieval module that operates on early LLM layers to identify and retain the most relevant pages, leveraging the model's inherent coarse-to-fine reasoning for evidence localization. The approach achieves state-of-the-art results across multiple long-document benchmarks while reducing computational overhead by 44-56% and introduces minimal parameter overhead. This end-to-end design eliminates external retrievers, enabling practical deployment and offering new insights into evidence-based reasoning in MLLMs.

Abstract

Recent multimodal large language models (MLLMs) still struggle with long document understanding due to two fundamental challenges: information interference from abundant irrelevant content, and the quadratic computational cost of Transformer-based architectures. Existing approaches primarily fall into two categories: token compression, which sacrifices fine-grained details; and introducing external retrievers, which increase system complexity and prevent end-to-end optimization. To address these issues, we conduct an in-depth analysis and observe that MLLMs exhibit a human-like coarse-to-fine reasoning pattern: early Transformer layers attend broadly across the document, while deeper layers focus on relevant evidence pages. Motivated by this insight, we posit that the inherent evidence localization capabilities of MLLMs can be explicitly leveraged to perform retrieval during the reasoning process, facilitating efficient long document understanding. To this end, we propose URaG, a simple-yet-effective framework that Unifies Retrieval and Generation within a single MLLM. URaG introduces a lightweight cross-modal retrieval module that converts the early Transformer layers into an efficient evidence selector, identifying and preserving the most relevant pages while discarding irrelevant content. This design enables the deeper layers to concentrate computational resources on pertinent information, improving both accuracy and efficiency. Extensive experiments demonstrate that URaG achieves state-of-the-art performance while reducing computational overhead by 44-56%. The code is available at https://github.com/shi-yx/URaG.

Figures (7)

• Figure 1: Comparison of different methods for long document understanding. Our method unifies retrieval and generation within a single MLLM, leveraging early-layer features for evidence retrieval during reasoning, which achieves efficient and accurate long document understanding.
• Figure 2: Analysis of MLLMs on long document understanding. (a) Attention entropy. (b) Attention-based retrieval accuracy. (c) Embedding-based retrieval accuracy.
• Figure 3: Overview of our URaG framework.
• Figure 4: Analysis of MLLMs on long document understanding. (a) Attention entropy. (b) Attention-based retrieval accuracy. (c) Embedding-based retrieval accuracy.
• Figure 5: Similarity heatmap for the query: "How many people are standing in front of the chalkboard in the photograph?".