Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan

TL;DR

Windsock addresses three core MRAG challenges: when to retrieve, what modality to retrieve, and how to utilize retrieved information. It introduces a query-driven Windsock module for adaptive retrieval, a Dynamic Noise-Resistance (DANCE) instruction tuning strategy, and a self-assessment data construction pipeline to create training signals without proprietary annotations. Empirical results on WebQA and MultimodalQA show Windsock with DANCE significantly improves generation quality and reduces retrieval time, outperforming strong baselines and some ground-truth-trained variants. The approach is scalable, modular, and extendable to additional modalities, offering practical impact for robust, efficient multimodal information retrieval in real-world applications.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.

Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

TL;DR

Windsock addresses three core MRAG challenges: when to retrieve, what modality to retrieve, and how to utilize retrieved information. It introduces a query-driven Windsock module for adaptive retrieval, a Dynamic Noise-Resistance (DANCE) instruction tuning strategy, and a self-assessment data construction pipeline to create training signals without proprietary annotations. Empirical results on WebQA and MultimodalQA show Windsock with DANCE significantly improves generation quality and reduces retrieval time, outperforming strong baselines and some ground-truth-trained variants. The approach is scalable, modular, and extendable to additional modalities, offering practical impact for robust, efficient multimodal information retrieval in real-world applications.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.
Paper Structure (50 sections, 5 equations, 8 figures, 19 tables)

This paper contains 50 sections, 5 equations, 8 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Retrieval-Augmented Generation
  4. Multimodal Large Language Models
  5. Method
  6. Task Definition
  7. Overview
  8. Windsock as an Adaptive Information Retriever
  9. Discussion
  10. Training Dataset Curation
  11. Discussion
  12. Dynamic Noise-Resistance Instruction Tuning
  13. Discussion
  14. Experiments
  15. Experimental Setup
  16. ...and 35 more sections

Figures (8)

  • Figure 1: Framework overview of our proposed method. Given a query, Windsock adaptively selects between direct answering (no retrieval) or retrieving from either visual or textual knowledge bases, followed by MLLM instruction tuned by DANCE, generating the final response based on the query and retrieved documents.
  • Figure 2: Data construction pipeline. An MLLM generates responses using different strategies (direct answering, visual retrieval, and textual retrieval), which are then evaluated to determine both the optimal retrieval strategy and challenging modality for training.
  • Figure 3: Windsock ratio of retrieval decisions on WebQA and WebQA+MS-COCO, showing the model's adaptive behavior across different complexities in downstream tasks.
  • Figure 4: Pipeline runtime breakdown of Qwen2-VL, showing the proportion of total inference time consumed by each component.
  • Figure 5: Performance scaling with backbone model size on WebQA, comparing F1 scores versus the number of parameters.
  • ...and 3 more figures