Self-adaptive Multimodal Retrieval-Augmented Generation

TL;DR

Self-adaptive Multimodal Retrieval-Augmented Generation (SAM-RAG) is proposed, tailored specifically for multimodal contexts and surpasses existing state-of-the-art methods in both retrieval accuracy and response generation.

Abstract

Traditional Retrieval-Augmented Generation (RAG) methods are limited by their reliance on a fixed number of retrieved documents, often resulting in incomplete or noisy information that undermines task performance. Although recent adaptive approaches alleviated these problems, their application in intricate and real-world multimodal tasks remains limited. To address these, we propose a new approach called Self-adaptive Multimodal Retrieval-Augmented Generation (SAM-RAG), tailored specifically for multimodal contexts. SAM-RAG not only dynamically filters relevant documents based on the input query, including image captions when needed, but also verifies the quality of both the retrieved documents and the output. Extensive experimental results show that SAM-RAG surpasses existing state-of-the-art methods in both retrieval accuracy and response generation. By further ablation experiments and effectiveness analysis, SAM-RAG maintains high recall quality while improving overall task performance in multimodal RAG task. Our codes are available at https://github.com/SAM-RAG/SAM_RAG.

Figures (6)

• Figure 1: Some questions rely on visual information, which make text-only retrieval unfeasible. These questions require retrieving and reasoning over visual context.
• Figure 2: The illustration of SAM-RAG pipeline. If the documents are visual, they will be converted to textual captions, vectorized and stored with textual documents. When a retrieval is performed, documents are processed in batch based on semantic similarity. If a caption is in a batch, its coressponding image will be recaptioned with the query.
• Figure 3: Effect of the Different Retrieval Numbers on Recall@N and EM.
• Figure 4: Complete real cases. (1) is a comparison between (a) convntional RAG and (b) SAM-RAG. (2) is a case where $\textbf{isUse}$ is False. (3) is a case where $\textbf{isSup}$ is Partial. For simplicity, only two visual documents are displayed.
• Figure A1: Image title: Kentucky Derby