E5-V: Universal Embeddings with Multimodal Large Language Models

TL;DR

The paper presents E5-V, a prompt-based framework that unifies multimodal embeddings within multimodal LLMs, enabling universal multimodal representations without fine-tuning. By training exclusively on text pairs and removing the visual encoder during training, it dramatically reduces data and compute needs while still delivering strong zero-shot performance on text-image, composed image, and image-image retrieval, as well as sentence embeddings. Extensive experiments demonstrate competitive or superior results across four task families, validating the approach's effectiveness and cost-efficiency. The work also reveals emergent zero-shot instruction-following capabilities and highlights the modality-gap elimination as a central enabler for transferring single-modality representations to multimodal contexts.

Abstract

Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection. Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality.

Figures (5)

• Figure 1: 2D visualization of multimodal embeddings and token embeddings in MLLM. Words correspond to the tokens in MLLM, and dots represent the multimodal embeddings. Our method unifies different multimodal embeddings from MLLM into the same space corresponding to their meanings without fine-tuning.
• Figure 2: Single modality training in E5-V. By unifying multimodal representations into the same embedding space with prompts, E5-V improves multimodal embeddings using only contrastive learning on text pairs. During training, we remove the modality encoder and projector in MLLM.
• Figure 3: Distribution of image embeddings and text embeddings from MLLM without and with our representation method.
• Figure 4: An example of rendering text as image with text "A man in a black shirt rides an elephant as a man walks near it down a street."
• Figure 5: Distribution of image embeddings and text embeddings with different representation methods.