Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

TL;DR

Unite72 presents a universal multimodal embedder that unifies text, image, and video representations through targeted data curation and a modality-aware training objective, MAMCL. The framework employs a two-stage training pipeline—retrieval adaptation and instruction tuning—using a LoRA-backed Qwen2-VL backbone and an evolving curriculum to maximize cross-modal retrieval performance. Key contributions include a detailed analysis of data composition effects, a novel MAMCL loss that isolates negatives by modality, and extensive experiments showing state-of-the-art results on 40+ tasks across coarse-grained, fine-grained, and instruction-based retrieval. The work provides practical guidelines for building robust universal multimodal systems and establishes a blueprint for future research in fused-modal information retrieval.

Abstract

Multimodal information retrieval (MIR) faces inherent challenges due to the heterogeneity of data sources and the complexity of cross-modal alignment. While previous studies have identified modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce UNITE, a universal framework that tackles these challenges through two critical yet underexplored aspects: data curation and modality-aware training configurations. Our work provides the first comprehensive analysis of how modality-specific data properties influence downstream task performance across diverse scenarios. Moreover, we propose Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive relationships among the instances of different modalities. Our framework achieves state-of-the-art results on multiple multimodal retrieval benchmarks, outperforming existing methods by notable margins. Through extensive experiments, we demonstrate that strategic modality curation and tailored training protocols are pivotal for robust cross-modal representation learning. This work not only advances MIR performance but also provides a foundational blueprint for future research in multimodal systems. Our project is available at https://friedrichor.github.io/projects/UNITE.

Figures (6)

• Figure 1: Performance comparison on instruction-based retrieval benchmarks (left: MMEB jiang2024vlm2vec and right: WebVid-CoVR ventura2024covr). Our Unite72, 203, 19472, 116, 203 achieves leading performance on various tasks, even surpassing models with larger parameter scales.
• Figure 2: Overview of Unite72, 203, 19472, 116, 203: (a) Model architecture utilizing LMM as the backbone, supporting multimodal inputs (text, images, videos, and their combinations). (b) Similarity matrix after applying MAMCL, which enables focused contrastive learning by restricting comparisons to samples sharing the same target modality, thus reducing inter-modal interference.
• Figure 3: We develop a universal multimodal embedder Unite72, 203, 19472, 116, 203, allowing for a unified representation of arbitrary multimodal contents.
• Figure 4: Performance comparison on fine-grained video-text benchmark (CaReBench xu2025carebench) and image-text benchmarks (ShareGPT4V chen2024sharegpt4v, Urban1K zhang2024longclip, DOCCI onoe2024docci). Our Unite72, 203, 19472, 116, 203 achieves the overall optimal performance.
• Figure 5: Retrieval Adaptation 6.4M. Left: Data Distribution within Each Category. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets.