Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

TL;DR

Zoom-shot enables fast, unsupervised transfer of CLIP zero-shot capabilities to any pre-trained vision encoder by learning a linear mapping $h: \mathbb{R}^m \to \mathbb{R}^d$ and its inverse $h^{-1}$ between the encoder latent spaces, guided by multimodal losses. The reconstruction loss $\mathcal{L}_{mse}$, variance-alignment, and a novel multimodal loss (cycle-consistency plus prompt-guided knowledge distillation, PG-KD) are used to capture interactions between text and image features in CLIP’s latent space, addressing the modality gap. Training is data-efficient and can be performed in a single epoch using unpaired data, with results showing consistent improvements over the Linear Aligner across coarse and fine-grained datasets and multiple encoders; LM-based PG-KD generally outperforms HT-CE, and the distribution of training data significantly affects transfer quality. This approach yields a practical, scalable pathway to extend CLIP-like capabilities to diverse vision encoders, with substantial compute/data trade-offs and accessible code and models available on GitHub; the method leverages $h$ and $h^{-1}$ to enable zero-shot classification without paired data or labels. $

Abstract

The fusion of vision and language has brought about a transformative shift in computer vision through the emergence of Vision-Language Models (VLMs). However, the resource-intensive nature of existing VLMs poses a significant challenge. We need an accessible method for developing the next generation of VLMs. To address this issue, we propose Zoom-shot, a novel method for transferring the zero-shot capabilities of CLIP to any pre-trained vision encoder. We do this by exploiting the multimodal information (i.e. text and image) present in the CLIP latent space through the use of specifically designed multimodal loss functions. These loss functions are (1) cycle-consistency loss and (2) our novel prompt-guided knowledge distillation loss (PG-KD). PG-KD combines the concept of knowledge distillation with CLIP's zero-shot classification, to capture the interactions between text and image features. With our multimodal losses, we train a $\textbf{linear mapping}$ between the CLIP latent space and the latent space of a pre-trained vision encoder, for only a $\textbf{single epoch}$. Furthermore, Zoom-shot is entirely unsupervised and is trained using $\textbf{unpaired}$ data. We test the zero-shot capabilities of a range of vision encoders augmented as new VLMs, on coarse and fine-grained classification datasets, outperforming the previous state-of-the-art in this problem domain. In our ablations, we find Zoom-shot allows for a trade-off between data and compute during training; and our state-of-the-art results can be obtained by reducing training from 20% to 1% of the ImageNet training data with 20 epochs. All code and models are available on GitHub.

Figures (5)

• Figure 1: A summary of our results comparing the average top-1 zero-shot test accuracy of the recent state-of-the-art in this problem domain, Linear Aligner t2c2023, to our proposed Zoom-shot method. The averaged results are from MobileNetV3 small Howard2019SearchingFM, DenseNet-121 Huang2016DenselyCC, ResNet-18 He2015DeepRL, DINOv1 (ViT-B/16) caron2021emerging and DINOv2 (ViT-B/14) dinov22023 vision encoders. We divide our testing datasets into coarse-grained (CIFAR-10/100 krizhevsky_learning_2009 and ImageNet-100/1000 deng_imagenet_2009) and fine-grained (CUB-200 WahCUB_200_2011, Flowers-102 Nilsback08, Herbarium-19 tan2019herbarium, Oxford-IIIT Pets parkhi12a and Stanford Cars 6755945). Our method consistently outperforms the Linear Aligner.
• Figure 2: A diagram summarizing our Zoom-shot method. Zoom-shot trains a mapping function $h$ and its inverse $h^{-1}$ for mapping between a vision encoder's latent space and CLIP's latent space. The resulting mapping functions can be used to transfer the zero-shot performance of CLIP to an arbitrary vision encoder. At its core, Zoom-shot consists of three loss functions: reconstruction loss, cycle-consistency loss and Prompt-Guided Knowledge Distillation. We only display some of the student variants for PG-KD.
• Figure 3: TSNE TSNE visualisation of all training datasets in the CLIP ViT-B/16 image encoders latent space. We can observe an interspersion of ImageNet features among the datasets like Oxford-IIIT pets and CUB-200; while at the same time there are differing degrees of separation for the Stanford Cars and Herbarium-19 datasets.
• Figure 4: TSNE TSNE visualization of pretrained image and text features from a randomly selected 10% of the MS-COCO dataset mscoco. The divide between the sets of features demonstrates the modality gap clearly present in CLIP clip2021.
• Figure 5: Prompts used for Prompt Guided Knowledge Distillation as generated by ChatGPT ChatGPT.