MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization

TL;DR

MLIP tackles CLIP’s data-inefficiency by introducing frequency-domain supervision via a Frequency Stage and joint spatial-frequency token alignment, enabling multi-domain and multi-level cross-modal learning. It further accelerates training with a controllable token merging mechanism guided by frequency-spatial cues, achieving a favorable balance between performance and compute. Empirically, MLIP improves zero-shot classification and image-text retrieval over CLIP baselines and sustains efficient training across multiple datasets and architectures, with benefits amplified by guide-driven merging. The work presents a practical approach to more data-efficient multimodal pretraining, leveraging frequency information to enrich supervision and token-level alignment.

Abstract

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success, leading to rapid advancements in multimodal studies. However, CLIP faces a notable challenge in terms of inefficient data utilization. It relies on a single contrastive supervision for each image-text pair during representation learning, disregarding a substantial amount of valuable information that could offer richer supervision. Additionally, the retention of non-informative tokens leads to increased computational demands and time costs, particularly in CLIP's ViT image encoder. To address these issues, we propose Multi-Perspective Language-Image Pretraining (MLIP). In MLIP, we leverage the frequency transform's sensitivity to both high and low-frequency variations, which complements the spatial domain's sensitivity limited to low-frequency variations only. By incorporating frequency transforms and token-level alignment, we expand CILP's single supervision into multi-domain and multi-level supervision, enabling a more thorough exploration of informative image features. Additionally, we introduce a token merging method guided by comprehensive semantics from the frequency and spatial domains. This allows us to merge tokens to multi-granularity tokens with a controllable compression rate to accelerate CLIP. Extensive experiments validate the effectiveness of our design.

Figures (7)

• Figure 1: (a) A distorted image. (b) An objective error map. The house and the sky regions are easily observable, and those on textural regions (e.g. rocks) are less noticeable, i.e., HVS is much more sensitive to the low-frequency variations than the high-frequency variations. (c) The original images and spectrums of the same lying cat in different scenes. It shows spectrum is pretty effective in extracting and differentiating features such as the complexity and noise of a scene (the high-frequency variations).
• Figure 2: The observation of performing similarity calculation and reduction operations on tokens on ImageNet. (a) Average token similarity in each layer of CLIP-ViT-B/32. (b) Zero-shot accuracy of random token reductions on different layers.
• Figure 3: The overall framework of MLIP. We modify the image encoder, and related design lies in the colorful areas and indexes.
• Figure 4: The process of Token Merging: Step1. Divide tokens at odd positions into set $A$ and those at even positions into set $B$. Step2. Find the most similar token in $B$ for each token in $A$ by calculating cosine similarity. Step3. Put similar tokens together to complete the match. Step4. Merge the similar tokens by weights.
• Figure 5: Text-to-image top 10 retrieval results on MS-COCO.