Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs

TL;DR

This work investigates Reverse Image Retrieval (RIR) as a simple, scalable way to augment multimodal LLMs with web-scale visual context to tackle knowledge-intensive VQA. By returning a screenshot of reverse image search results as contextual input, RIR substantially improves performance for GPT-4 family models and moderately benefits open models, while revealing that improvements often arise from better alignment between visual queries and the model's parametric world knowledge rather than from direct retrieval of factual answers. The analysis shows RIR provides strongest gains on long-tail concepts and objects, with some scenarios where it can hurt, and demonstrates that a naive always-on-RIR baseline can outperform a model that must decide when to use RIR. The findings highlight the potential of web-scale multimodal memory to enhance factual grounding and imply the need for robust decision policies when integrating retrieval-augmented mechanisms into future MLLM systems.

Abstract

Despite impressive advances in recent multimodal large language models (MLLMs), state-of-the-art models such as from the GPT-4 suite still struggle with knowledge-intensive tasks. To address this, we consider Reverse Image Retrieval (RIR) augmented generation, a simple yet effective strategy to augment MLLMs with web-scale reverse image search results. RIR robustly improves knowledge-intensive visual question answering (VQA) of GPT-4V by 37-43%, GPT-4 Turbo by 25-27%, and GPT-4o by 18-20% in terms of open-ended VQA evaluation metrics. To our surprise, we discover that RIR helps the model to better access its own world knowledge. Concretely, our experiments suggest that RIR augmentation helps by providing further visual and textual cues without necessarily containing the direct answer to a query. In addition, we elucidate cases in which RIR can hurt performance and conduct a human evaluation. Finally, we find that the overall advantage of using RIR makes it difficult for an agent that can choose to use RIR to perform better than an approach where RIR is the default setting.

Figures (6)

• Figure 1: Overview of the reverse image retrieval (RIR) augmented generation pipeline. In this example, GPT-4V was used as the MLLM backbone. Calling RIR is as easy as running the following line of Python code: rir_api.query_with_image(image_url, query_text). In this basic example leveraging a rare bird species, the correct answer is contained in the search results which may not be the case for knowledge-intensive problems that go beyond the identification of the displayed object.
• Figure 2: Selected Output Examples of GPT-4 Turbo before and after being augmented by reverse image retrieval (RIR). RIR helps identify objects and provide relevant information for unique objects or sites. We also find that in a few cases such as more general and widely-available concepts like certain animals or foods, RIR may be detrimental. Overall, we find that RIR robustly improves the ability of GPT-4-level MLLMs to answer knowledge-intensive visual questions (Section \ref{['sec:main_results']}).
• Figure 3: Illustration of reverse image retrieval (RIR) pipeline. We discover that when prompted with knowledge-intensive visual questions, state-of-the-art MLLMs like GPT-4o can fail to leverage their own world knowledge. Augmenting the query with multimodal RIR results improves the vision-language alignment and allows extraction of highly-specialized text-based knowledge from the model.
• Figure 4: MLLMs provided with text-only questions that contain oracle-provided entities (brown) show high accuracy on INFOSEEK, showcasing that MLLMs do possess the factual knowledge required, but cannot leverage it in Vanilla VQA prompt. RIR helps close the gap (especially in GPT-4 type models).
• Figure 5: Samples where RIR helps (green) have fewer Google search results compared to hurting set (brown), supporting the hypothesis that entities with less online presence are underrepresented in MLLM training datasets, thus benefiting more from RIR's capabilities.