Re-ranking the Context for Multimodal Retrieval Augmented Generation
Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus
TL;DR
This work addresses the challenge of selecting relevant context in multimodal retrieval-augmented generation by replacing or augmenting CLIP-based retrieval with a learned relevancy score (RS) model. The RS is trained to assess query-context relevance and is used to re-rank retrieved items, enabling adaptive selection of the final context size. By combining an initial fast CLIP-based pass with RS-driven re-ranking and threshold-based pruning, the approach improves both the relevance of retrieved data and the factual quality of generated responses, as demonstrated on COCO-derived datasets. The study reports substantial improvements in relevancy and output accuracy, highlights the potential to reduce hallucinations in multimodal RAG, and provides the RS/CS resources for further research via the mortahebRAGCheck project.
Abstract
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-$k$ entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.