GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

TL;DR

This work tackles universal multimodal retrieval by proposing GME, a dense embedding framework built on Multimodal LLMs that can process text, images, visual documents, and fused content. It introduces a large-scale, carefully balanced data composition and a novel fused-modal data synthesis pipeline to overcome fused-modal data scarcity, achieving state-of-the-art results on the new UMR Benchmark (UMRB). The paper provides extensive analyses on data mix, model scaling, and training strategies, and releases UMRB (and UMRB-Partial) to benchmark universal multimodal retrieval. Overall, GME demonstrates modality-universal embeddings and strong cross-modal and fused-modal retrieval, highlighting the viability of unified MLLM-based retrievers for diverse modalities. The work also discusses practical considerations, such as instruction-tuning and hard-negative mining, that contribute to performance gains across tasks.

Abstract

Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data.

Figures (10)

• Figure 1: Illustration of different retrieval settings in our universal multimodal retrieval task. Blocks with black borders represent data in arbitrary modalities, i.e. text-only, image-only or fused.
• Figure 2: The GME model architecture. Emb denotes the embedding of the input content.
• Figure 3: Impact of training data on multimodal retrieval tasks.
• Figure 4: Pipeline for synthesizing fused-modal training data.
• Figure 5: Visualization of the embeddings in a 2D plot by T-SNE. Left: Our GME, Middle: VISTA, Right: CLIP. We use instances from Encyclopedia VQA and highlight two semantic groups with yellow and pink labels, respectively. Please zoom in to view them.