MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin

TL;DR

This work introduces MARVEL, a Multi-modAl Retrieval model via Visual modulE pLugin, which unifies the encoding of queries and multi-modal documents by integrating a visual module with a pre-trained dense retriever. The model uses a universal encoder, T5-ANCE-CLIP, to embed text and image documents into a shared space and employs image-caption contrastive pretraining followed by modality-balanced finetuning to align modalities. The authors also contribute ClueWeb22-MM, a large-scale multi-modal benchmark built from anchor-linked web pages, enabling evaluation of truly multi-modal retrieval. Empirical results show MARVEL achieves state-of-the-art performance on WebQA and ClueWeb22-MM, with analyses demonstrating the importance of the visual module pretraining, image captions, and a plugin-based fusion strategy. Overall, MARVEL demonstrates how text retrieval knowledge can be extended to multi-modal scenarios by bridging the modality gap with a plug-in visual module and a unified embedding space.

Abstract

This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVEL encodes queries and multi-modal documents with a unified encoder model, which helps to alleviate the modality gap between images and texts. Specifically, we enable the image understanding ability of the well-trained dense retriever, T5-ANCE, by incorporating the visual module's encoded image features as its inputs. To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and extracts the related text and image documents from anchor-linked web pages. Our experiments show that MARVEL significantly outperforms the state-of-the-art methods on the multi-modal retrieval dataset WebQA and ClueWeb22-MM. MARVEL provides an opportunity to broaden the advantages of text retrieval to the multi-modal scenario. Besides, we also illustrate that the language model has the ability to extract image semantics and partly map the image features to the input word embedding space. All codes are available at https://github.com/OpenMatch/MARVEL.

Figures (4)

• Figure 1: Retrieval Pipeline with Our MARVEL Model. MARVEL incorporates the visual module plugin, aiming to unlock the multi-modal capabilities of well trained dense retrieval model.
• Figure 2: The Architecture of Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL). We first pretrain the visual modules using the image-caption alignment task (Figure \ref{['fig:model:ptetrain']}) and then finetune the language model to conduct multi-modal retrieval (Figure \ref{['fig:model:finetune']}).
• Figure 3: Attention Distribution of MARVEL-ANCE. The attention weights of image features are shown in Figure \ref{['fig:embed:weight']}. And the attention weight entropy of image captions and features is shown in Figure \ref{['fig:embed:entropy']}.
• Figure 4: Case Studies. We present two cases from WebQA and ClueWeb22-MM and show the top5 retrieved multi-modal documents. The ground-truth documents and related content are highlighted in red and blue respectively.