Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learning

Ruoshuang Du, Xin Sun, Qiang Liu, Bowen Song, Zhongqi Chen, Weiqiang Wang, Liang Wang

TL;DR

Experiments demonstrated that the proposed Multimodal Adaptive RAG effectively balances external knowledge utilization and inference robustness in diverse multimodal scenarios and achieves a significant improvement in response performance in three VQA datasets.

Abstract

Visual Question Answering systems face reliability issues due to hallucinations, where models generate answers misaligned with visual input or factual knowledge. While Retrieval Augmented Generation frameworks mitigate this issue by incorporating external knowledge, static retrieval often introduces irrelevant or conflicting content, particularly in visual RAG settings where visually similar but semantically incorrect evidence may be retrieved. To address this, we propose Multimodal Adaptive RAG (MMA-RAG), which dynamically assesses the confidence in the internal knowledge of the model to decide whether to incorporate the retrieved external information into the generation process. Central to MMA-RAG is a decision classifier trained through a layer-wise analysis, which leverages joint internal visual and textual representations to guide the use of reverse image retrieval. Experiments demonstrated that the model achieves a significant improvement in response performance in three VQA datasets. Meanwhile, ablation studies highlighted the importance of internal representations in adaptive retrieval decisions. In general, the experimental results demonstrated that MMA-RAG effectively balances external knowledge utilization and inference robustness in diverse multimodal scenarios.

Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learning

TL;DR

Experiments demonstrated that the proposed Multimodal Adaptive RAG effectively balances external knowledge utilization and inference robustness in diverse multimodal scenarios and achieves a significant improvement in response performance in three VQA datasets.

Abstract

Visual Question Answering systems face reliability issues due to hallucinations, where models generate answers misaligned with visual input or factual knowledge. While Retrieval Augmented Generation frameworks mitigate this issue by incorporating external knowledge, static retrieval often introduces irrelevant or conflicting content, particularly in visual RAG settings where visually similar but semantically incorrect evidence may be retrieved. To address this, we propose Multimodal Adaptive RAG (MMA-RAG), which dynamically assesses the confidence in the internal knowledge of the model to decide whether to incorporate the retrieved external information into the generation process. Central to MMA-RAG is a decision classifier trained through a layer-wise analysis, which leverages joint internal visual and textual representations to guide the use of reverse image retrieval. Experiments demonstrated that the model achieves a significant improvement in response performance in three VQA datasets. Meanwhile, ablation studies highlighted the importance of internal representations in adaptive retrieval decisions. In general, the experimental results demonstrated that MMA-RAG effectively balances external knowledge utilization and inference robustness in diverse multimodal scenarios.
Paper Structure (13 sections, 4 equations, 6 figures, 3 tables)

This paper contains 13 sections, 4 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Datasets and Knowledge Bases
  5. Backbone Models and Metrics
  6. Baselines
  7. Results
  8. Main Results
  9. Feature Robustness
  10. Ablation Study
  11. RIR Strategy Comparison
  12. Conclusion
  13. References

Figures (6)

  • Figure 1: Overview of the Multimodal Adaptive RAG framework
  • Figure 2: Heatmap of the classifier accuracy across different key tokens and layers on the OK-VQA dataset using the Idefics2-8B model
  • Figure 3: Impact of Transformer Layer Selection on Visual Feature Representation
  • Figure 4: Layer-wise Comparison of Classifier Accuracy Using Textual and Multimodal Features on the InfoSeek, OK-VQA, and E-VQA Datasets
  • Figure 5: Layer-wise Comparison of Classifier Accuracy with Textual and Visual Features on OK-VQA Dataset
  • ...and 1 more figures