SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information

TL;DR

This paper tackles the instability of retrieval-augmented large vision-language models by introducing SURf, a self-refinement framework that teaches LVLMs to selectively utilize retrieved image-caption information. SURf constructs positive and negative examples from retrieved content based on whether they improve answers to initially incorrect questions, filters these examples, and performs RAG instruction-tuning with retrieved context enclosed in explicit tags. Through experiments on seven datasets covering VQA, image captioning, and image classification, SURf achieves significant gains over baselines and demonstrates robustness to irrelevant or misleading retrieval content. The approach highlights the value of hard-negative sampling and data filtering in enabling reliable multimodal RAG, suggesting a practical path toward more trustworthy LVLMs.

Abstract

Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf.

Figures (11)

• Figure 1: Illustration of multimodal RAG. RAG can introduce misleading content, causing LVLMs to generate incorrect responses. SURf can selectively utilize information from images and descriptions, e.g., the first image-caption pair.).
• Figure 2: The illustration of Multimodal RAG for VQA, Captioning and Classification Tasks. Providing images similar to the test images along with their descriptions as references can help LVLMs answer questions more accurately.
• Figure 3: Illustration of our training framework. First, we collect questions that LVLMs initially answered incorrectly. Next, we retrieve the Top-N image-caption pairs from the corpus, allowing the LVLM to reattempt the questions. We then evaluate the answers to see if they have improved (positive) or worsened (negative). After that, we filter for the highest-quality training data and use it for instruction tuning to train the LVLMs.
• Figure 4: Performance of the base model (LLaVA-1.5-7B) without using RAG (Base), RAG with irrelevant content (Irrelevant), and RAG on POPE-popular, MS-COCO, and CIFAR-10.
• Figure 5: Ablation Study of Database Size and Data Filter.