UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning

TL;DR

The paper tackles the limitations of in-batch negative mining in universal multimodal embedding by introducing UniME-V2, which leverages Multimodal Large Language Models as judges to generate soft semantic matching scores for diverse, high-quality hard negatives and distribution-aligned learning. It further adds UniME-V2-Reranker to refine initial retrieval with joint pairwise and listwise optimization. Through global retrieval-based hard-negative mining and soft-label supervision, UniME-V2 achieves state-of-the-art averages across the MMEB benchmark and multiple retrieval tasks, including compositional and long-caption retrieval. The work demonstrates robust improvements over strong baselines and provides detailed ablations and implementation specifics to support reproducibility and further research in universal multimodal representation learning.

Abstract

Universal multimodal embedding models are foundational to various tasks. Existing approaches typically employ in-batch negative mining by measuring the similarity of query-candidate pairs. However, these methods often struggle to capture subtle semantic differences among candidates and lack diversity in negative samples. Moreover, the embeddings exhibit limited discriminative ability in distinguishing false and hard negatives. In this paper, we leverage the advanced understanding capabilities of MLLMs to enhance representation learning and present a novel Universal Multimodal Embedding (UniME-V2) model. Our approach first constructs a potential hard negative set through global retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs and generate soft semantic matching scores. These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives. Furthermore, the semantic matching scores are used as soft labels to mitigate the rigid one-to-one mapping constraint. By aligning the similarity matrix with the soft semantic matching score matrix, the model learns semantic distinctions among candidates, significantly enhancing its discriminative capacity. To further improve performance, we propose UniME-V2-Reranker, a reranking model trained on our mined hard negatives through a joint pairwise and listwise optimization approach. We conduct comprehensive experiments on the MMEB benchmark and multiple retrieval tasks, demonstrating that our method achieves state-of-the-art performance on average across all tasks.

Figures (7)

• Figure 1: Comparison between previous works and UniME-V2. UniME-V2 exploits the understanding capabilities of MLLMs for hard negatives mining and generates a soft semantic matching score to supervise the model in learning the semantic difference among candidates.
• Figure 2: The MLLM-as-a-Judge pipeline for Hard Negatives Mining. We first utilize an existing multimodal embedding model for global retrieval to construct a potential hard negative set. We then leverage the powerful understanding capabilities of MLLM to score query-candidate pairs based on their semantic alignment, enabling precise identification of hard negatives.
• Figure 3: The architecture of the MLLM Judgment Based Training Framework. UniME-V2 uses soft semantic matching scores as supervised signals to enhance semantic distinction learning between candidates. UniME-V2-Reranker employs joint pairwise and listwise optimization to enhance reranking performance.
• Figure 4: Comparison of representation distributions between EVA-CLIP-8B and UniME-V2 (LLaVA-OneVision-7B).
• Figure 5: Qualitative examples. We present the retrieval and reranking results of our method across different tasks.