Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization

TL;DR

The paper tackles the information bottleneck and limited semantic coverage inherent in single-embedding multimodal representations by introducing the Parallel Decoupling Framework (PDF). PDF uses deep prefix injection to create $N$ parallel paths within an MLLM, generating $N$ distinct embeddings per input and enforcing diversity with Mutual Information Minimization (MIM) via the vCLUB estimator, all while maintaining semantic alignment through per-path and aggregated InfoNCE losses. Empirically, PDF yields consistent gains across backbones (e.g., Qwen2VL and LLaVA-1.6) and resolutions on the MMEB benchmark, with notable improvements such as +8.9 points for 7B-scale low-resolution models and +12.1 points for certain 2B configurations, plus substantial training efficiency gains. Importantly, inference can proceed with a single path, incurring negligible overhead, and zero-shot retrieval improvements demonstrate strong generalization. Overall, PDF advances multimodal embedding by enabling diversified, robust representations without sacrificing efficiency.

Abstract

Embedding models are a cornerstone of modern AI. Driven by Multimodal Large Language Models (MLLMs), they have made great progress in architecture and data curation, while the holistic paradigm is still limited to SSC, i.e., single input, singular embedding, contrastive supervision, which collapses rich, multifaceted inputs into monolithic embeddings and fails to fully exploit MLLM capabilities. In this paper, we tailor one Parallel Decoupling Framework (PDF) for multimodal embedding learning, by utilizing the proprietary steerability of MLLMs, i.e., their ability to flexibly generate quite differentiated response under explicit instructions. Concretely, PDF conditions a shared MLLM backbone on distinct, learnable prefixes to roll out multiple parallel paths for one input, then relies on these paths to obtain parallel embeddings. To promote full parallel diversity, we employ Mutual Information Minimization (MIM) as an explicit constraint, coupled with per-path contrastive supervision to maintain semantic alignment. Such dual-objectives force PDF to yield robust semantic coverage and a generalizable embedding space. Ultimately, the remarkable embedding space are accessible at inference via one single forward pass, incurring negligible computational overhead. We instantiate PDF on multiple MLLM backbones and prove its effectiveness on MMEB benchmark. Significant gains are consistently achieved across various resolutions and model sizes, e.g., boosting the VLM2Vec-LLaVA-1.6-LR model by a remarkable +8.9% (7B), while the VLM2Vec-Qwen2VL models by +4.2% (2B) and +3.1% (7B). In terms of efficiency, our 2B model surpasses its baseline by +2.6% using only half the computational budget.

Figures (5)

• Figure 1: Comparisons of Framework & Performance. (a) Previous: the ubiquitous SSC framework, i.e., single input, singular embedding, contrastive supervision. (b) Parallel Decoupling Framework (PDF): single input rolls out as parallel inputs, then generates multiple embeddings explicitly de-correlated by Mutual Information Minimization (MIM). (c) On the VLM2Vec-Qwen2VL backbone, our PDF consistently delivers significant gains across diverse tasks and model scales.
• Figure 2: Overview of the PDF-VLM2Vec Pipeline. The training process (blue and green lines) is guided by a dual-objective system. For each input, parallel embeddings are generated via learnable prefixes. These are then supervised by: (1) Contrastive Loss to maintain representation quality, and (2) MIM Loss to enforce diversity. The MIM loss is calculated in a two-stage process: first updating the MI estimator with detached embeddings (Stage 1), and then using the frozen estimator to compute the loss (Stage 2). During inference (red line), a single forward pass inherits these benefits, yielding a robust embedding with additional negligible computational overhead.
• Figure 3: Record of the training dynamics on VLM2Vec-Qwen2VL (2B).Left: Training loss vs. baseline. Right: Cosine similarity of parallel prefixes w/ and w/o MIM loss.
• Figure 4: Qualitative Results Part 1. We show the results of our method across six different retrieval tasks compared with VLM2Vec-Qwen2VL-7B.
• Figure 5: Qualitative Results Part 2. We show the results of our method across six different retrieval tasks compared with VLM2Vec-Qwen2VL-7B.