CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

TL;DR

CatLIP tackles the computational bottleneck of CLIP-style pre-training by reframing image-text pre-training as a multi-label classification task using WordNet synsets derived from captions. It achieves about $2.7\times$ faster pre-training on web-scale data while preserving CLIP-level transfer accuracy across ImageNet-1k, Places365, and downstream tasks like detection and segmentation. The approach scales data and model size, enabling data-efficient transfer learning via classifier-embedding transferInit and achieving competitive results with standard CLIP-based methods on diverse tasks. The work provides open-source code and weights to enable broader adoption of efficient image-text pre-training on noisy web data.

Abstract

Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text embeddings. However, pairwise similarity computation in contrastive loss between image and text pairs poses computational challenges. This paper presents a novel weakly supervised pre-training of vision models on web-scale image-text data. The proposed method reframes pre-training on image-text data as a classification task. Consequently, it eliminates the need for pairwise similarity computations in contrastive loss, achieving a remarkable $2.7\times$ acceleration in training speed compared to contrastive learning on web-scale data. Through extensive experiments spanning diverse vision tasks, including detection and segmentation, we demonstrate that the proposed method maintains high representation quality. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}.

Figures (8)

• Figure 1: CatLIP is $\mathbf{2.7\times}$ faster to pre-train than CLIP while maintaining down-stream accuracy. For a fair comparison, we calculate GPU hours by training CatLIP and CLIP with a batch size of 65k for one epoch of DataComp-1.3B on the same hardware configuration. Finetuning accuracy of CatLIP and CLIP is reported on ImageNet-1k dataset. Here, represents trainable backbones.
• Figure 2: ImageNet-1k labels appear frequently in image-text datasets, contributing to increased zero-shot classification accuracies of CLIP models. (a) shows the process of finding labels of the target dataset in the image-text dataset. (b) shows the number of ImageNet-1k synsets at different similarity thresholds between synset vocabulary and ImageNet-1k synset. Exact match (similarity score of 1.0) for approximately 40% of ImageNet-1k labels is found in the DataComp-1.3B captions. (c) illustrates occurrences of ImageNet-1k synsets in image-text datasets, with larger datasets exhibiting more samples per synset. Here, the vocabulary pruning threshold, $V_\tau$, is set to $500$.
• Figure 3: Analysis of extracted WordNet synsets in image-text datasets. Larger datasets typically contain a greater number of synsets, indicating increased content diversity in larger datasets.
• Figure 4: CatLIP vs. CLIP. Unlike CLIP, CatLIP benefits from longer training on CC3M dataset and requires significantly less pre-training computation. Here, (a) shows the top-1 linear probe accuracy of CLIP and CatLIP on ImageNet-1k as a function of CC3M pre-training epochs while (b) shows the linear probe accuracy as a function of pre-training duration. Here, each dot in a graph represents an independently trained model on CC3M. See \ref{['sec:app_training_details']} for training details.
• Figure 5: Linear probe transfer learning with CatLIP is more data efficient. Here, transfer learning is achieved by training a linear classifier on downstream tasks for 30 epochs with frozen image backbone features (each dot in a graph is an independent run). Linear classifier is initialized either using a standard method (Random Init) or transferred classifier embeddings from a pre-trained model (Transfer Init; ours). Initializing classifier with Transfer Init delivers better accuracy than Random Init, especially in small data regime.