VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

TL;DR

The paper tackles the gap in universal multimodal embeddings by introducing MMEB, a large-scale benchmark that spans four meta-tasks and 36 datasets, and Vlm2Vec, a contrastive framework that converts vision-language models into instruction-following embedders. Vlm2Vec employs instruction-guided contrastive learning with an InfoNCE objective and scales training with GradCache, achieving state-of-the-art results across MMEB and strong zero-shot generalization to out-of-distribution data. The best variant (LLaVA-1.6 backbone with LoRA and high-resolution inputs) reaches 62.9% average precision@1 across all MMEB tasks and 57.1% on OOD tasks, outperforming baselines by up to ~18 points. The work demonstrates that vision-language models can serve as powerful embedding backbones when trained on diverse tasks with instruction guidance and efficient training techniques, paving the way for universal multimodal representations.

Abstract

Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite its importance and practicality. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets covering both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-Language Model -> Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, which encodes text or images independently without any task instruction, VLM2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB. We show that VLMs are secretly strong embedding models.

Figures (5)

• Figure 1: We develop a universal multimodal embedding benchmark, MMEB, along with Vlm2Vec, an embedding model adapted from vision-language models (VLMs). Vlm2Vec is capable of following instructions and performing various multimodal embedding tasks, accommodating any combination of image and text modalities.
• Figure 2: An overview of the tasks and datasets in MMEB. MMEB includes four meta-tasks and 36 datasets: 20 in-distribution datasets (blue) used for training and 16 out-of-distribution (orange) datasets used exclusively for evaluation.
• Figure 3: Vlm2Vec uses a VLM as the backbone to deeply integrate image and text features. It is trained with a contrastive loss between the query and target, following task-specific instructions. The training data consists of diverse combinations of modalities on both the query and target sides, which may include images, text, or image-text pairs.
• Figure 4: The figures demonstrate the influence of the training setup on Vlm2Vec's final performance. Here, we examine the effects of training batch size, the number of sub-image crops, and the number of training steps. All the models utilize Phi-3.5-V as their backbone.
• Figure 5: The figures show the generalization ability of models trained on one meta-task to other unseen meta-tasks. For example, the first subplot compares the performance of Vlm2Vec trained exclusively on VQA datasets with Vlm2Vec trained exclusively on retrieval datasets across the other two meta-task categories: classification and visual grounding. Overall, Vlm2Vec trained on retrieval tasks demonstrate better generalization ability because retrieval tasks involve a more diverse combination of text and visual modalities from both the query and target sides. Vlm2Vec utilizes Phi-3.5-V as its backbone.