A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

TL;DR

The paper probes which elements of language supervision are essential for zero-shot image classification by replacing natural captions with Bag-of-Words and balancing word frequencies. It introduces a cross-modal contrastive pre-training framework and a BoW deformation pipeline, then extends training with pseudo-BoW captions for unaligned images in a semi-aligned setting. Empirical results show BoW can match or surpass intact captions across multiple architectures, and semi-aligned learning with 3M unaligned images attains zero-shot accuracy on par with CLIP-15M. The findings highlight data-efficient supervision strategies for cross-modal learning, with practical implications for reducing reliance on richly labeled captions and enabling scalable, robust zero-shot vision models.

Abstract

Using natural language as a supervision for training visual recognition models holds great promise. Recent works have shown that if such supervision is used in the form of alignment between images and captions in large training datasets, then the resulting aligned models perform well on zero-shot classification as downstream tasks2. In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models. Through extensive and careful experiments, we show that: 1) A simple Bag-of-Words (BoW) caption could be used as a replacement for most of the image captions in the dataset. Surprisingly, we observe that this approach improves the zero-shot classification performance when combined with word balancing. 2) Using a BoW pretrained model, we can obtain more training data by generating pseudo-BoW captions on images that do not have a caption. Models trained on images with real and pseudo-BoW captions achieve stronger zero-shot performance. On ImageNet-1k zero-shot evaluation, our best model, that uses only 3M image-caption pairs, performs on-par with a CLIP model trained on 15M image-caption pairs (31.5% vs 31.3%).

Figures (6)

• Figure 1: Overview of our method: We train image and text encoders with a cross-modal contrastive loss, similar to convirtclip, with image and caption pairs, but we design various operators to deform the intact caption. For example, shuffling the words, removing certain words, etc. Colors except green, represents words removed by different deformation operators. The resulting caption is a Bag-of-Words with only a few words from the original caption. We show that a model trained with BoW captions performs on-par or better than the model trained with intact captions (29.5% vs 30.1% on ImageNet-1k zero-shot).
• Figure 2: From intact captions to BoW. Above figure illustrates the process of converting intact captions to BoW. Notice that BoW are much denser with very little distracting information. The words are first shuffled to remove syntax by Shuffle. Next, while stop words ("all", "were", "the", etc.) and non-alphabetical tokens (":" and ".") are important parts of the syntax of sentences, none of them describe anything in their corresponding images. Hence, they are removed with RmStopNalpha. Next, words like "cars", "table", "men", "rocks" etc. (orange color) are important for describing the image, but because they already appear frequently we can remove them with RmTopFreq(1000). Note that the model still sees and learns about them but from the small set of captions that are kept intact (base set). Further, the blue word "mikoshi" is a rare word that does not appear in the base set vocabulary, and hence maybe removed without hurting the model. It is removed by LimitToBaseVocab. Purple words like "utter" and "ash" are removed by Keep(n) which may actually remove useful words, yet BoW improve the performance. Finally, some captions will become empty since none of the words in them describe anything new. Thus, they are removed from the dataset.
• Figure 3: Intact vs. BoW on additional datasets. We compare intact and default BoW caption based models with 4 different architectures on zero-shot evaluation across 8 different datasets. The results show that for some datasets like, Flowers102, DTD, and Pets, all BoW models improve regardless of the architecture. While, for some of the datasets like, Caltech101, Sun397, and Food101, the accuracy of the deformed model consistently decreases. See the supplementary material for a detailed analysis of the results.
• Figure 4: Task hardness vs. BoW. We hypothesize that removing frequent words makes the task harder since the model cannot rely on co-occurrence statistics of the frequent words to solve the task of distinguishing correct image-caption pairs during pre-training. The loss curves in the top figure shows evidence that supports our hypothesis. The curve with highest loss uses the default deformations to create BoW where 1k most frequent words are removed (BoW without frequent words). Unlike other deformations, it shifts the focus towards less-frequent concepts in the text. This is shown in the bottom figure where the IN-1k-ZS accuracy quickly converges to its final accuracy for intact captions, but it takes longer and reaches higher for the default deformation (BoW without frequent words). The deformation which keeps the frequent words (BoW with frequent words) simply makes the task harder but the model can still use frequent words to solve the task without learning other concepts.
• Figure 5: Generating pseudo BoW captions: We illustrate the process of constructing pseudo BoW captions for unaligned images in this figure. A fully aligned dataset, CC conceptual_captions, to train the vision and text encoders and to obtain the intact captions used for retrieval. We find nearest neighbors in the text domain for different crops of a given unaligned image. The retrieved captions of all crops are then aggregated into a BoW. The words are then ranked according to one of the strategies mentioned in Section 2.3 of main text, and a few top words (4 in this figure) are chosen as the pseudo BoW caption. We used the WeightedCount strategy in this figure.