Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

TL;DR

The paper investigates why CLIP-style image-language models exhibit strong robustness to natural distribution shifts. Through controlled experiments isolating training set size, data distribution, language supervision, test-time prompts, and loss functions, the authors find that diverse training distributions are the key driver of robustness, with language supervision contributing little direct benefit. By introducing ImageNet-Captions and conducting comprehensive YFCC-based studies, they demonstrate that image-only pretraining on diverse data can match CLIP-like robustness, underscoring the primacy of data-centric factors over language cues. The work emphasizes dataset design as a crucial lever for improving real-world robustness in multimodal models.

Abstract

Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts. Since these language-image models differ from previous training approaches in several ways, an important question is what causes the large robustness gains. We answer this question via a systematic experimental investigation. Concretely, we study five different possible causes for the robustness gains: (i) the training set size, (ii) the training distribution, (iii) language supervision at training time, (iv) language supervision at test time, and (v) the contrastive loss function. Our experiments show that the more diverse training distribution is the main cause for the robustness gains, with the other factors contributing little to no robustness. Beyond our experimental results, we also introduce ImageNet-Captions, a version of ImageNet with original text annotations from Flickr, to enable further controlled experiments of language-image training.

Figures (12)

• Figure 1: We compare models trained using different methods and on different datasets, measuring their robustness on a range of natural distribution shifts (ImageNetV2, ImageNet-R, ImageNet-Sketch, and ObjectNet). The CLIP models stand out with their consistent performance in the presence of distribution shift. We find that large gains in effective robustness (improvement over ImageNet models) only come from varying the training distribution. Language supervision alone does not cause robustness.
• Figure 2: Overview of the two main training sets in our experiments. (Top) We introduce the ImageNet-Captions dataset, where we augment a subset of the ImageNet 2012 training set images with the corresponding original captions collected from Flickr. (Bottom) We convert the YFCC image-caption dataset into YFCC-Classification by searching for class labels in the YFCC captions and then removing the text annotations. These two datasets allows us to evaluate the impact of language-image training on robustness because we can compare language-image training with standard classification training on the same set of images.
• Figure 3: Three sample images from ImageNet-Captions. Their respective ImageNet labels are: drake, rocking chair, payphone.
• Figure 4: On most natural distribution shifts, models trained with language information from ImageNet-Captions follow the same trend as models trained without it. Neither comes close to achieving the robustness of OpenAI's CLIP models.
• Figure 5: Using the weights from OpenAI’s pre-trained CLIP model does not improve robustness, despite the large size of the full CLIP training set (400 million images). This is further evidence that language supervision does not increase robustness. See Figure \ref{['fig:imagenet_captions_base']} for remaining legend elements.