LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning

TL;DR

This work tackles the challenge of distinguishing hard negative pairs in large multimodal models trained with InfoNCE, showing that standard contrastive learning yields overlapping positive and negative similarity distributions. It introduces a hardness-weighted contrastive learning framework plus cross-device negative sampling, where a reward model dynamically weights negatives to emphasize harder cases, and negatives are gathered across devices to enlarge the pool. The LLaVE family (0.5B, 2B, 7B) demonstrates state-of-the-art performance on the MMEB benchmark across 4 meta-tasks and 36 datasets, with notable gains in in-distribution and out-of-distribution settings, and it transfers effectively to zero-shot text-video retrieval. These results indicate improved discriminative learning for multimodal embeddings, scalable training, and strong potential for broader transfer to other embedding tasks including video modalities.

Abstract

Universal multimodal embedding models play a critical role in tasks such as interleaved image-text retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.

Figures (5)

• Figure 1: Similarity distributions of learned embeddings on the SUN397 SUN397 and RefCOCO RefCOCO dataset. We present the query-target cosine similarity histograms of positive, hard negative, and easy negative pairs for the model trained with the standard InfoNCE loss and LLaVE.
• Figure 2: Overview of hardness-weighted contrastive learning. Please note that the policy and reward models are identical in our work. The dashed line indicates directly copying the parameters of the policy model to the reward model.
• Figure 3: An example of cross-device negative sample gathering ($N$=$4$ and $K$=$3$). The plus signs represent positive pairs, and the minus signs represent negative pairs. Each device calculates the similarity between its own queries and the targets on all other devices, which is then used for loss computation.
• Figure 4: Influence of the $\alpha$ on model performance, measured on IND and OOD datasets, respectively.
• Figure 5: Qualitative evaluation comparing LLaVE and VLM2Vec. Retrievals consistent with the ground truth are highlighted with red borders. From left to right, the images represent the top-1 to top-3 retrieval results.