Visual RAG: Expanding MLLM visual knowledge without fine-tuning

TL;DR

This work tackles the problem of outdated knowledge and limited context in Multimodal Large Language Models by proposing Visual RAG, a retrieval-augmented framework that dynamically expands visual knowledge without fine-tuning. It integrates an image-based knowledge base with CLIP embeddings and FAISS indexing to retrieve highly relevant visual demonstrations, which are then provided to a Gemini-based generator via a carefully designed prompt. Across eight diverse datasets, Visual RAG matches or exceeds the performance of many-shot in-context learning while using substantially fewer demonstrations (about 23% on average), and even achieves notable efficiency gains by reducing context size and computation. The approach enables rapid updates to visual capabilities and scalable expansion into new domains, with future work aimed at refining retrieval methods and extending to other computer vision tasks.

Abstract

Multimodal Large Language Models (MLLMs) have achieved notable performance in computer vision tasks that require reasoning across visual and textual modalities, yet their capabilities are limited to their pre-trained data, requiring extensive fine-tuning for updates. Recent researches have explored the use of In-Context Learning (ICL) to overcome these challenges by providing a set of demonstrating examples as context to augment MLLMs performance in several tasks, showing that many-shot ICL leads to substantial improvements compared to few-shot ICL. However, the reliance on numerous demonstrating examples and the limited MLLMs context windows presents significant obstacles. This paper aims to address these challenges by introducing a novel approach, Visual RAG, that synergically combines the MLLMs capability to learn from the context, with a retrieval mechanism. The crux of this approach is to ensure to augment the MLLM knowledge by selecting only the most relevant demonstrating examples for the query, pushing it to learn by analogy. In this way, relying on the new information provided dynamically during inference time, the resulting system is not limited to the knowledge extracted from the training data, but can be updated rapidly and easily without fine-tuning. Furthermore, this greatly reduces the computational costs for improving the model image classification performance, and augments the model knowledge to new visual domains and tasks it was not trained for. Extensive experiments on eight different datasets in the state of the art spanning several domains and image classification tasks show that the proposed Visual RAG, compared to the most recent state of the art (i.e., many-shot ICL), is able to obtain an accuracy that is very close or even higher (approx. +2% improvement on average) while using a much smaller set of demonstrating examples (approx. only 23% on average).

Figures (5)

• Figure 1: Schematic representation of the architecture of Visual RAG: the query image and the Knowledge Base (KB) constituted by image-label pairs are fed to the embedding model to obtain their vector representations. In the embedding spaces the retriever returns the requested number of demonstrating examples by selecting the images in the KB whose vectors are closer to the vector of the query image. The generator (i.e., MLLM) is then prompted with the query image and the demonstrating examples to predict the label for the query image.
• Figure 2: Examples of images in each dataset considered. From top to bottom: HAM10000, FIVES, CheXpert, TerraIncognita, UCMerced, EuroSAT, Oxfort Pets, DTD. Images are reshaped to square size for a more compact visualization.
• Figure 3: Accuracy comparison between the proposed Visual RAG solution, Many-shot ICL yixing2024manyshot, and zero-shot baseline on the eight datasets considered.
• Figure 4: Efficiency comparison between the proposed Visual RAG solution and Many-shot ICL yixing2024manyshot. For each dataset, a comparison is made between the experiment with the highest accuracy obtained using Many-shot ICL, and the experiment with the lowest number of examples that was able to match or surpass that accuracy using Visual RAG. In cases where the accuracy was not improved, the experiment exhibiting the closest performance was selected. The plot reports the accuracy gain of Visual RAG with respect to Many-shot ICL, versus the ratio of demo samples used.
• Figure 5: Example results of the proposed Visual RAG in the experimental configuration where five demonstration examples are provided by the retriever. Each row represents a different case. For each case we report: the query image, the five retrieved examples, the answer provided by Visual RAG with the corresponding confidence score, and the ground truth label. From top to bottom the query images belong to: Oxford Pets, UCMerced, DTD, and EuroSAT dataset.