CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning

TL;DR

This work tackles data quality for multimodal contrastive pretraining by introducing two universal data-filtering methods that do not rely on external models. The surrogate-CLIPLoss (s-CLIPLoss) refines CLIPScore by normalizing with the contrastive batch loss, reducing biases from dataset quirks, while NormSim leverages downstream task distribution via vision-only embeddings to measure training-target similarity. On the DataComp benchmark, s-CLIPLoss and NormSim yield substantial gains over CLIPScore (e.g., 5.3% on ImageNet-1k and 2.8% across 38 tasks) and are compatible with top external methods, pushing toward state-of-the-art results when combined with DFN and HYPE. The approach is model-agnostic and can be applied with various CLIP backbones, offering practical improvements in data filtering efficiency and downstream performance, with a theoretical justification for NormSim under linearity assumptions and a proxy-target extension (NormSim_2-D) when task data are inaccessible.

Abstract

Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric). While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce surrogate-CLIPLoss (s-CLIPLoss), a CLIP loss-inspired method that adds the alignment between one sample and its contrastive pairs as an extra normalization term for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, NormSim, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp~\cite{gadre2023datacomp}. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\% improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation tasks. Moreover, both s-CLIPLoss and NormSim are compatible with existing techniques. By combining our methods with the current best methods DFN and HYPE, we can boost average performance on downstream tasks by 0.9\%, achieving a new state-of-the-art on the DataComp-medium benchmark.

Figures (11)

• Figure 1: Illustration of s-CLIPLoss. CLIPScore may underestimate (bottom left, where the data quality is high but CLIPScore is low (negative CLIPScore is high)) or overestimate (bottom right, where the data quality is low but CLIPScore is high (negative CLIPScore is low)) the quality of image-text pairs. However, this issue can be mitigated by simply including a normalization term $\mathcal{R}$. s-CLIPLoss employs the teacher model to calculate the surrogate CLIP loss on training data and serves as a more accurate metric. Here, "Bottom X%" denotes that the score represents the bottom X% low values within the entire dataset (i.e., the X% percentile among all the values). For example, "$\mathcal{R}: \text{Bottom} \ 0\%$" means this data has almost the smallest $\mathcal{R}$ among the whole dataset, which represents that it contains highly specific elements in both images and texts. The lower X in s-CLIPLoss should correspond to data with higher quality.
• Figure 2: s-CLIPLoss consistently outperforms CLIPScore across different downsampling ratios on DataComp-medium.
• Figure 3: Illustration of NormSim.$X_\text{target}$ is the target prior data. "Top X%" denotes that the score represents the top X% high values within the entire dataset. (a) Visualization of data with different NormSim and s-CLIPLoss. Here we use $\text{NormSim}_2$(ImageNet-1k) as an example. Although both Type 2 and Type 4 data have high s-CLIPLoss and thus high quality, data with low $\text{NormSim}_2$ (Type 4) are more irrelevant to downstream tasks like ImageNet, VTAB, and MSCOCO. For example, they contain many images dominated by OCR content and make little contribution to improving downstream performance. (b) Illustration of a rough comparison of sampling data for different filtering methods. Using "$\text{s-CLIPLoss} \cap \text{NormSim}$" filtering can balance the quality and relevance to downstream tasks, thus increasing the proportion of Type 2 data. (Refer to Appendix \ref{['supp: add_vis']} for more visualization.)
• Figure 4: Illustration of different directions for data selection methods for multimodal contrastive learning. Here we use four colors to denote the four main resources we can obtain: CLIP teacher model, downstream target data (which is much smaller than the external multimodal dataset or pretraining dataset), the external image-text dataset, and the external non-CLIP model. Direction 1 denotes the methods that only use the original OAI CLIP teacher model and the downstream target data. Direction 2 represents the methods that use external datasets to train a new CLIP teacher model for improving filtering, like DFN fang2023data. Direction 3 denotes the methods that use external non-CLIP model to select the data that may be heuristically helpful for downstream tasks, like image without too much text or be more special. In general, D1 method using only CLIP embedding, like s-CLIPLoss, is orthogonal to D2. And both D1 and D2 can be combined with D3 to explore better filtering results. In the experiments part of the main paper (Sec. \ref{['sec: experiment']}), we further show that our proposed D1 methods: NormSim and s-CLIPLoss, can outperform all the D3 baselines except the best method "HYPE $\cup$ DFN". And we can achieve the new state-of-the-art by combining our methods with that method.
• Figure 5: Results of s-CLIPLoss with a different number of batch samples (denoted as $K$) on DataComp-medium. Solid lines denote s-CLIPLoss, while dashed lines denote CLIPScore. Here, we use OAI CLIP-L/14 as the pretrained model. We can see that once $K \geq 5$, s-CLIPLoss consistently outperforms CLIPScore across all subtask metrics. In the main paper, we set $K=10$.

Theorems & Definitions (15)

• Theorem A.1
• proof
• Lemma A.1: Intuition behind $\text{NormSim}_2$
• proof : Proof sketch
• Theorem A.2: Main
• Lemma A.2
• proof
• Lemma A.3
• proof
• Lemma A.4: Intuition behind VAS