CMRAG: Co-modality-based visual document retrieval and question answering

TL;DR

The findings of this paper show that integrating co-modality information into the RAG framework in a unified manner is an effective approach to improving the performance of complex VDQA systems.

Abstract

Retrieval-Augmented Generation (RAG) has become a core paradigm in document question answering tasks. However, existing methods have limitations when dealing with multimodal documents: one category of methods relies on layout analysis and text extraction, which can only utilize explicit text information and struggle to capture images or unstructured content; the other category treats document segmentation as visual input and directly passes it to visual language models (VLMs) for processing, yet it ignores the semantic advantages of text, leading to suboptimal retrieval and generation results. To address these research gaps, we propose the Co-Modality-based RAG (CMRAG) framework, which can simultaneously leverage texts and images for more accurate retrieval and generation. Our framework includes two key components: (1) a Unified Encoding Model (UEM) that projects queries, parsed text, and images into a shared embedding space via triplet-based training, and (2) a Unified Co-Modality-informed Retrieval (UCMR) method that statistically normalizes similarity scores to effectively fuse cross-modal signals. To support research in this direction, we further construct and release a large-scale triplet dataset of (query, text, image) examples. Experiments demonstrate that our proposed framework consistently outperforms single-modality--based RAG in multiple visual document question-answering (VDQA) benchmarks. The findings of this paper show that integrating co-modality information into the RAG framework in a unified manner is an effective approach to improving the performance of complex VDQA systems.

Figures (8)

• Figure 1: Comparison among (a) text--based RAG, (b) image--based RAG, and (c) co-modality--based RAG.
• Figure 2: An overview of the proposed CMRAG framework. (a) A VLM is prompted to parse visual documents offline. (b) Images, parsed texts, and given queries are encoded uniformly in a shared space. Images and texts can be encoded and indexed offline to accelerate the online RAG systems. (c) The calculated similarity scores of visual and textual modalities are unified to a comparable distribution, ensuring a more accurate retrieval. (d) A VLM generator is prompted to generate the final answer based on the query and retrieved evidence.
• Figure 3: Unified distributions of query-image (Sim-I) and query-text (Sim-T) similarity scores of (a) Finslides, (b) Tehslides, and (c) LongDocURL.
• Figure 4: Prompt templates for (a) parsing images, (b) generating answers based on entire images, (c) generating answers based on sub-images and text, generating answers based on entire images and text, and (e) judging generated answers. The first template can be found at https://github.com/QwenLM/Qwen2.5VL/blob/main/cookbooks/document_parsing.ipynb and the rest can be referred to wang2025vrag.
• Figure 5: training process.