TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

TL;DR

This work introduces a novel contrastive pre-training strategy that leverages hard negative captions and images in an alternating fashion to train CLIP, and demonstrates that this method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement on the SugarCrepe benchmark on an equal computational budget.

Abstract

Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io

Figures (7)

• Figure 1: Comparison of training workflows of CLIP, NegCLIP, and TripletCLIP. $(x, y)$ represents the positive a image-text pair, and ($x^\prime, y^\prime$) represents the corresponding negative image-text pair.
• Figure 2: Examples image-text pairs from TripletData. In each block, a positive pair from CC3M is on the left and corresponding negatives from TripletData are shown on the right.
• Figure 3: Average Results of LaCLIP and TripletCLIP for SugarCrepe Compositions, Image-Text Retrieval, and ImageNet1k over increasing concept diversity.
• Figure 4: Positive vs. Negative modality-specific pair-based similarity distribution of pre-trained CLIP ViT-B/32 model w.r.t. the vision and text-only encoders. The left plot is the vision embedding similarities between positive and negative images. The right plot is the text embedding similarities between positive and negative captions. In the ideal scenario, the distribution should be skewed towards 0.0, which indicates that the model can correctly distinguish between the positive and negative data.
• Figure 5: Positive vs. Negative modality-specific pair-based similarity distribution of baseline LaCLIP and TripletCLIP. The left plot is the vision embedding similarities between positive and negative images. The right plot is the text embedding similarities between positive and negative captions.