Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Philip S. Yu, Fei Huang, Jingren Zhou

TL;DR

The paper argues that fixed, non-adaptive retrieval in multimodal RAG systems leads to non-adaptive and overloaded queries,欠 and introduces Dyn-VQA to benchmark dynamic, multimodal knowledge retrieval. It proposes OmniSearch, a self-adaptive planning agent that decomposes complex multimodal questions into sub-questions and iteratively retrieves information using diverse tools. Empirical results show OmniSearch, especially with GPT-4V as the sub-question solver, substantially improves over heuristic mRAG baselines and commercial generative search engines, though challenges remain on fast-changing and high-hop scenarios. The work provides a new dataset and a plug-and-play retrieval framework that advances robust, adaptive multimodal QA and offers open-source resources for the community.

Abstract

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

TL;DR

The paper argues that fixed, non-adaptive retrieval in multimodal RAG systems leads to non-adaptive and overloaded queries,欠 and introduces Dyn-VQA to benchmark dynamic, multimodal knowledge retrieval. It proposes OmniSearch, a self-adaptive planning agent that decomposes complex multimodal questions into sub-questions and iteratively retrieves information using diverse tools. Empirical results show OmniSearch, especially with GPT-4V as the sub-question solver, substantially improves over heuristic mRAG baselines and commercial generative search engines, though challenges remain on fast-changing and high-hop scenarios. The work provides a new dataset and a plug-and-play retrieval framework that advances robust, adaptive multimodal QA and offers open-source resources for the community.

Abstract

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

Paper Structure

This paper contains 41 sections, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Retrieval Augmented Generation
  4. Dyn-VQA Dataset
  5. Dyn-VQA Construction
  6. Dyn-VQA Analysis
  7. Retrieval Baselines and OmniSearch
  8. Baselines
  9. OmniSearch Framework
  10. Experiments
  11. Experimental Settings
  12. Backbone MLLMs for heuristic mRAGs.
  13. Evaluation Metric.
  14. Main Results
  15. Analysis Experiments on OmniSearch
  16. ...and 26 more sections

Figures (8)

  • Figure 1: Bottom: Heuristic mRAG based VQA. Upper: OmniSearch based VQA.
  • Figure 2: Dynamic VQA examples that require different kinds of retrieval strategies.
  • Figure 3: Left: overall framework of OmniSearch. Right: running example of OmniSearch.
  • Figure 4: Model performance of different domains.
  • Figure 5: Pairwise overlap between correctly answered questions of different models. 0-8: Qwen-VL-Chat, Qwen-7B-Chat, Deepseek-VL-7B-Chat, VisualGLM-6B, Llava-V1.6-Mistral-7B, mPLUG-Owl2.1, InstructBLIP-Vicuna-7B, Qwen-VL-Max, GPT-4V.
  • ...and 3 more figures