Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications

TL;DR

Results reveal that multimodal RAG can outperform single-modality RAG settings, although image retrieval poses a greater challenge than text retrieval, and leveraging textual summaries from images presents a more promising approach compared to the use of multimodal embedded models.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in answering questions, but they lack domain-specific knowledge and are prone to hallucinations. Retrieval Augmented Generation (RAG) is one approach to address these challenges, while multimodal models are emerging as promising AI assistants for processing both text and images. In this paper we describe a series of experiments aimed at determining how to best integrate multimodal models into RAG systems for the industrial domain. The purpose of the experiments is to determine whether including images alongside text from documents within the industrial domain increases RAG performance and to find the optimal configuration for such a multimodal RAG system. Our experiments include two approaches for image processing and retrieval, as well as two LLMs (GPT4-Vision and LLaVA) for answer synthesis. These image processing strategies involve the use of multimodal embeddings and the generation of textual summaries from images. We evaluate our experiments with an LLM-as-a-Judge approach. Our results reveal that multimodal RAG can outperform single-modality RAG settings, although image retrieval poses a greater challenge than text retrieval. Additionally, leveraging textual summaries from images presents a more promising approach compared to the use of multimodal embeddings, providing more opportunities for future advancements.

Figures (11)

• Figure 1: Overall architecture of our proposed multimodal RAG pipelines. For the Text-Only RAG, we only use the Text Retrieval component. Conversely, in the Image-Only RAG, we only employ the Image Retrieval component, either with multimodal embeddings or image summaries.
• Figure 2: RAG evaluation results for GPT-4V prompted with either a single or multiple images, and LLaVA (always single image) across six metrics. The results show the performance of each RAG setting in generating accurate, relevant, and faithful responses based on both text and image inputs.
• Figure 3: Question Answering Prompt.
• Figure 4: Image Summarization Prompt.
• Figure 5: Answer Correctness Evaluation Prompt.