UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings

TL;DR

UniMoCo addresses the challenge of aligning diverse modality combinations in multi-modal embeddings by introducing a modality-completion module that generates pseudo visual embeddings from text and a training strategy that jointly optimizes contrastive alignment and cross-modal consistency. Using a LVLM backbone, padding-enabled modality completion, and an auxiliary loss, it achieves robust performance across classification, VQA, retrieval, and grounding on MMEB, with the Qwen2-VL-7B variant achieving an average of $63.2$ and reducing biases toward dominant modality combinations. Ablation and bias analyses confirm the importance of padding, an auxiliary alignment term, and architecture choices in maintaining cross-modality coherence. The results suggest practical impact for robust multi-modal retrieval and reasoning in real-world settings where modality availability is inconsistent.

Abstract

Current research has explored vision-language models for multi-modal embedding tasks, such as information retrieval, visual grounding, and classification. However, real-world scenarios often involve diverse modality combinations between queries and targets, such as text and image to text, text and image to text and image, and text to text and image. These diverse combinations pose significant challenges for existing models, as they struggle to align all modality combinations within a unified embedding space during training, which degrades performance at inference. To address this limitation, we propose UniMoCo, a novel vision-language model architecture designed for multi-modal embedding tasks. UniMoCo introduces a modality-completion module that generates visual features from textual inputs, ensuring modality completeness for both queries and targets. Additionally, we develop a specialized training strategy to align embeddings from both original and modality-completed inputs, ensuring consistency within the embedding space. This enables the model to robustly handle a wide range of modality combinations across embedding tasks. Experiments show that UniMoCo outperforms previous methods while demonstrating consistent robustness across diverse settings. More importantly, we identify and quantify the inherent bias in conventional approaches caused by imbalance of modality combinations in training data, which can be mitigated through our modality-completion paradigm. The code is available at https://github.com/HobbitQia/UniMoCo.

Figures (4)

• Figure 1: (a) Multi-modal embedding tasks involve three common modality combinations, sampled from the MMEB benchmark jiang2024vlm2vec: $(\text{T}+\text{I},\text{T})$, $(\text{T},\text{T}+\text{I})$, and $(\text{T}+\text{I},\text{T}+\text{I})$. Specifically, $(\text{T}+\text{I},\text{T})$ represents tasks where the query consists of both text and image modalities, while the target only includes text. The other combinations, $(\text{T},\text{T}+\text{I})$ and $(\text{T}+\text{I},\text{T}+\text{I})$, can be interpreted in an analogous manner. A multi-modal embedding model encodes both the query and the target into a unified embedding space, and conduct tasks such as information retrieval, visual grounding, etc, by similarity matching. (b) UniMoCo's performance vs. other embedding models on MMEB benchmark.
• Figure 2: (a) UniMoco architecture. Processes image/text inputs through an LLM, with the final output token as the unified embedding. (b) UniMoCo workflow illustration. The left panel shows image-text processing while the right panel shows text-only input processing. Grayed-out icons indicate inactive modules in each scenario. This unified workflow supports both training and inference phases.
• Figure 3: UniMoCo training strategy. The approach combines a primary contrastive loss ($\mathcal{L}_1$) with an auxiliary loss term ($\mathcal{L}_2$) to optimize model performance. Black lines indicate positive pairs to be pulled closer in the embedding space, while red lines denote negative pairs requiring separation.
• Figure 4: Analysis of modality combination biases arising from skewed training data distributions. The x-axis labels (1, 2, 3) correspond to training sets dominated by $(\text{T}+\text{I},\text{T})$, $(\text{T},\text{T}+\text{I})$, and $(\text{T}+\text{I},\text{T}+\text{I})$ combinations respectively.