Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

TL;DR

This work addresses knowledge-based visual question answering with multimodal LLMs by enabling dynamic external-knowledge integration. It introduces ReflectiVA, which augments a pretrained MLLM with reflective tokens that gate retrieval and judge the relevance of retrieved passages, trained through a two-stage, two-model pipeline. Across Encyclopedic-VQA and InfoSeek, ReflectiVA achieves state-of-the-art results and demonstrates robust generalization, while preserving performance on standard MLLM benchmarks. The approach advances retrieval-augmented multimodal reasoning and provides practical benefits for leveraging current knowledge sources in vision-language tasks, with public code and models available at the project site.

Abstract

Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data. They have recently garnered attention due to their capability to address complex tasks involving both modalities. However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility. In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources. Our proposed model, Reflective LLaVA (ReflectiVA), utilizes reflective tokens to dynamically determine the need for external knowledge and predict the relevance of information retrieved from an external database. Tokens are trained following a two-stage two-model training recipe. This ultimately enables the MLLM to manage external knowledge while preserving fluency and performance on tasks where external knowledge is not needed. Through our experiments, we demonstrate the efficacy of ReflectiVA for knowledge-based visual question answering, highlighting its superior performance compared to existing methods. Source code and trained models are publicly available at https://aimagelab.github.io/ReflectiVA.

Figures (7)

• Figure 1: Overview of ReflectiVA, which employs reflective tokens for knowledge-based visual question answering. Our model learns to predict the need of retrieving data from an external knowledge source (top), classifies the relevance of each retrieved item (middle) and generate the final answer based on relevant items (bottom).
• Figure 2: Training approach for ReflectiVA. An in-article model is trained to predict the relevance of passages extracted from the ground-truth document corresponding to an $(I, q)$ pair. The in-article model then generates training data for ReflectiVA, which is trained to predict the need for external knowledge and the relevance of passages, along with the answer, using positive, soft- and hard-negative passages.
• Figure 3: Sample qualitative results on image-question pairs from Encyclopedic-VQA (top row) and InfoSeek (bottom row), where we compare the answers provided by ReflectiVA with those from WikiLLaVA caffagni2024wiki and EchoSight yan2024echosight.
• Figure 4: Re-ranking performance on the single-hop question split of Encyclopedic-VQA when varying the number $k$ of retrieved documents and the number $k_p$ of passages given to our model after re-ranking. The "without re-ranking" bars refer to the best results achieved without using the re-ranking component, as reported in Table \ref{['tab:results']} of the main paper.
• Figure 5: Additional qualitative results on image-question pairs from Encyclopedic-VQA, where we compare the answers provided by ReflectiVA with those from WikiLLaVA caffagni2024wiki and EchoSight yan2024echosight.