Compression then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding

TL;DR

The paper addresses the data efficiency gap in converting vision-language models into strong multimodal embeddings by decoupling comprehensive input understanding from discriminative matching. It introduces CoMa, a compression pre-training stage that uses a small set of learnable compression tokens and an attention-mask mechanism to extract rich, image-wide information before engaging in contrastive learning. AQA-based compression objective $P(A \mid I \oplus \mathcal{C}(X) \oplus Q; \theta)$ and automatic data generation with multi-turn QA enable robust pretraining with limited data, followed by InfoNCE-based contrastive optimization. Empirically, CoMa achieves state-of-the-art or near-state-of-the-art results on the Massive Multimodal Embedding Benchmark (MMEB) with substantially less pretraining data and resource usage, highlighting improved efficiency and competitive effectiveness for downstream cross-modal retrieval and related tasks. The work also provides thorough ablations on compression token count, data formats, and loss choices, offering practical guidance for building data-efficient multimodal embedding systems with large-language-model backbones such as Qwen2.5-VL and LoRAfinetuning.

Abstract

Vision-language models advance multimodal representation learning by acquiring transferable semantic embeddings, thereby substantially enhancing performance across a range of vision-language tasks, including cross-modal retrieval, clustering, and classification. An effective embedding is expected to comprehensively preserve the semantic content of the input while simultaneously emphasizing features that are discriminative for downstream tasks. Recent approaches demonstrate that VLMs can be adapted into competitive embedding models via large-scale contrastive learning, enabling the simultaneous optimization of two complementary objectives. We argue that the two aforementioned objectives can be decoupled: a comprehensive understanding of the input facilitates the embedding model in achieving superior performance in downstream tasks via contrastive learning. In this paper, we propose CoMa, a compressed pre-training phase, which serves as a warm-up stage for contrastive learning. Experiments demonstrate that with only a small amount of pre-training data, we can transform a VLM into a competitive embedding model. CoMa achieves new state-of-the-art results among VLMs of comparable size on the MMEB, realizing optimization in both efficiency and effectiveness.

Figures (6)

• Figure 1: Architecture of CoMa, the top part demonstrates three training stages: (a) Instruction-Tuning, (b) our Continual PreTraining and (c) Contrastive Learning. The figures show: In (a) and (b), our continual pretraining stage inherits the same format of Question-Answering (QA) task as the upstream stage, while the answer in the upstream depends on the whole image, and our stage depends on the Compression Tokens which condense the image. In (c), the contrastive learning stage inherits the compression tokens as representations of multimodal input to apply for upstream retrieval-based tasks. The bottom part shows the implementation of our continual pretraining stage: (d) shows the model architecture where all the inputs are concatenated to calculate simultaneously, and the dependencies are driven by our modified causal attention masks in (e).
• Figure 2: The impact of different numbers of compressed tokens on performance.
• Figure 3: Similarity between compression tokens with different numbers: (a) 16, (b) 32, and (c) 64.
• Figure 4: Representations of queries and targets across Three Stages: (a) Base, (b) Compression Pretraining, and (c) Contrastive Learning. Representations in (a) are extracted from the base model via the [EOS] token, while the others are via compression tokens. The representations are decomposed using Principal Component Analysis (PCA).
• Figure 5: Loss distribution across tokens for Cross-Entropy (Top) and KL Divergence (Bottom).