MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

TL;DR

MuRAG addresses open-question answering over images and text by introducing a multimodal retrieval-augmented transformer that accesses an external memory containing images, text, or image-text pairs. It combines a ViT–T5 backbone with a retrieval mechanism (Top-K via MIPS) and a joint contrastive plus generative training objective, using an efficient in-batch memory during pre-training and a two-stage fine-tuning pipeline. On WebQA and MultimodalQA, MuRAG achieves state-of-the-art accuracy, outperforming text-only and multimodal baselines by 10–20 percentage points in both distractor and full-wiki settings, and ablations show the contributions of memory, pre-training data, and training strategy. The work demonstrates the feasibility and value of retrieval-augmented generation over multimodal knowledge and highlights areas for improvement in grounding for image-centric questions and scalability of visual representations during decoding.

Abstract

While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images -- much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20\% absolute on both datasets and under both distractor and full-wiki settings.

Figures (9)

• Figure 1: Visual information-seeking queries: These queries are unanswerable with text-only retrieval and require retrieving and reasoning over images.
• Figure 2: Model Overview: retrieval-and-predict process of MuRAG on downstream datasets.
• Figure 3: Backbone encoder: ViT encodes image patches into a sequence of vectors $\mathbf{e}_I$, while word embedding converts text tokens into another sequence of vectors $\mathbf{e}_T$. These vectors are concatenated to form $f_{\theta}(e)$ and fed to a decoder for text generation.
• Figure 4: Model Architecture: the model accesses an external memory to obtain multimodal knowledge contained in images or text snippets, which is used to augment the generation. The upper part defines the pre-training implementation, while the lower part defines fine-tuning implementation.
• Figure 5: Upper left: correct prediction, Upper Right: error due to miscounting, Lower: error due to misrecognition (multiple image reasoning). Q refers to the question, P refers to prediction and R refers to the reference.