Sieve: Multimodal Dataset Pruning Using Image Captioning Models

TL;DR

Sieve tackles the challenge of pruning noisy web-crawled image-text data used to train Vision-Language Models by introducing a caption-based alignment signal. It generates multiple captions with a pretrained captioning model, masks non-visual phrases, and uses a lightweight sentence transformer to measure semantic similarity between generated captions and alt-text, forming a robust alignment score $f_{\text{Sieve}}(I,T)$. When fused with CLIPScore, Sieve achieves improved average performance across 38 downstream tasks on DataComp, particularly enhancing retrieval, while ablations reveal the benefits of curated caption pretraining data, captioning models, and the sentence-embedding space. Overall, Sieve provides a complementary, data-efficient pruning signal that reduces false positives/negatives and enhances zero-shot multimodal performance with practical impact on large-scale dataset curation.

Abstract

Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these datasets is strongly correlated with the performance of VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train models using highly-aligned samples is one of the most successful methods for pruning. We argue that this approach suffers from multiple limitations including: false positives and negatives due to CLIP's pretraining on noisy labels. We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs. To bridge the gap between the limited diversity of generated captions and the high diversity of alternative text (alt-text), we estimate the semantic textual similarity in the embedding space of a language model pretrained on unlabeled text corpus. Using DataComp, a multimodal dataset filtering benchmark, when evaluating on 38 downstream tasks, our pruning approach, surpasses CLIPScore by 2.6\% and 1.7\% on medium and large scale respectively. In addition, on retrieval tasks, Sieve leads to a significant improvement of 2.7% and 4.5% on medium and large scale respectively.

Figures (7)

• Figure 1: Examples of image-text pairs in which the scores of pretrained CLIP models, a commonly used image-text data filtering approach, fail to measure their alignment. Our proposed approach, Sieve, provides an accurate alignment score using a caption generator and sentence transformer. Top left and bottom left: Examples of false positives where alt-text describes concepts that are not found or unrelated to the image. CLIP is trained on similar noisy image-text pairs, thus, it assigns a relatively high score. Sieve can detect that such image-text pairs are misaligned. Top right and bottom right: Examples of false negatives where images are aligned with the alt-text but are assigned low CLIP scores, either due to the low likelihood of these pairs in the pretraining data, or because CLIP may have seen similar images aligned with other noisy labels. Sieve can detect that such image-text pairs are well-aligned and selects them for pretraining.
• Figure 2: Our proposed framework enables dataset pruning using image-captioning models. To evaluate the alignment of a noisy image-text pair, we generate multiple captions per image using nucleus sampling. Then medium phrases, like "an image of" or "a photo of", are masked from alt-text and generated captions. Finally, a lightweight sentence encoder is used to semantically compare the generated captions with alt-text.
• Figure 3: Masking medium phrases improves the ranking of sentence similarity scores. On the left, sentence pairs with misleadingly high (or low) sentence similarity due to the existence (or absence) of medium phrases are highlighted in dark red (or light red). On the right, similarity scores that are more aligned with semantics are highlighted in dark green. The sentence similarity scores are computed using the all-MiniLM-L6-v2 sentence transformer wang2020minilm.
• Figure 4: Evaluating CLIP models pretrained on different fractions of the top ranked samples based on our proposed approach (Sieve), CLIPScore, and fusing Sieve with CLIPscore (Sieve+CLIPSCore), on medium scale.
• Figure 5: The relative performance gain of Sieve and Sieve+CLIPScore relative to CLIPScore on 38 downstream tasks on the medium scale pool.