Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification

TL;DR

This work addresses the mismatch between CLIP's cross-modal pretraining and unimodal feature extraction by proposing CODER, a cross-modal neighbor representation that leverages the distance between images and their text neighbors in CLIP's feature space. CODER constructs image representations from multiple text neighbors, enhanced by Auto Text Generator which creates diverse, high-quality texts without data or training, enabling dense sampling of semantics. The method is applied to zero-shot and few-shot CLIP classification, including a two-stage zero-shot pipeline with one-to-one reranking and a CODER-Adapter for few-shot scenarios; experiments show consistent improvements across datasets and model families, with ablations emphasizing the importance of text diversity and sampling density. The approach offers practical gains for deploying CLIP in low-data settings, while also revealing limitations related to text generation costs and the dimensionality of CODER, guiding future refinements in dense semantic sampling and reranking strategies.

Abstract

CLIP showcases exceptional cross-modal matching capabilities due to its training on image-text contrastive learning tasks. However, without specific optimization for unimodal scenarios, its performance in single-modality feature extraction might be suboptimal. Despite this, some studies have directly used CLIP's image encoder for tasks like few-shot classification, introducing a misalignment between its pre-training objectives and feature extraction methods. This inconsistency can diminish the quality of the image's feature representation, adversely affecting CLIP's effectiveness in target tasks. In this paper, we view text features as precise neighbors of image features in CLIP's space and present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. This feature extraction method aligns better with CLIP's pre-training objectives, thereby fully leveraging CLIP's robust cross-modal capabilities. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. We introduce the Auto Text Generator(ATG) to automatically generate the required texts in a data-free and training-free manner. We apply CODER to CLIP's zero-shot and few-shot image classification tasks. Experiment results across various datasets and models confirm CODER's effectiveness. Code is available at:https://github.com/YCaigogogo/CVPR24-CODER.

Figures (5)

• Figure 1: Illustration of image's CrOss-moDal nEighbor Representation ( Coder). CLIP's powerful text-image matching capabilities endows it with a favorable cross-modal neighbor distance relation. And CLIP's Zero-Shot Image Classification process can be interpreted as using a $1$NN algorithm to find the image's nearest text, with the text's class determining the image's predicted class. Inspired by this idea, we expand the image's neighbor range to leverage its distance to all texts for constructing the CODER. Here $d_{ij}$ refers to the distance between the $i$-th image and the $j$-th text.
• Figure 2: An example of CODER correcting wrong distance relation between images.$d_{ij}^{img}$ refers to the cosine distance between the $i$-th and the $j$-th image. $r_1$ and $r_2$ refer to the cosine distance between the text and different images, while $1-r_1$ and $1-r_2$ refer to the cosine similarity. The left side of the figure indicates that even though images of the same class share similar distance to a text, this doesn't ensure that their features are closely similar. The right side of the figure shows that CODER corrects the wrong distance relation by utilizing the text-image distance.
• Figure 3: Illustration of two-stage zero-shot image classification process based on image's CODER. In the first stage, we use the Auto Text Generator to create a General Text Set, which contains general descriptions of classes. This set is utilized to construct the image's general CODER, and we use it for preliminary classification. In the second stage, we construct One-to-One Text Sets in pairs for the top five predicted classes of the preliminary classification results, focusing on attributes where the two specific classes differ most. We build one-to-one specific CODER for the image based on these One-to-One Text Sets and use heuristic classifier to get each class's classification score. Then we rerank the top five preliminary results based on the classification score gaps $gap_i^{cj}$ between classes. Here $gap_i^{cj}$ represents the difference obtained by subtracting the score of the class $c$ from that of the class $j$ for image ${\boldsymbol{x}}_i$.
• Figure 4: Results for the training-free few-shot regime across 11 datasets. We compare the CODER-Adapter with the previous CLIP few-shot image classification methods. Our CODER-Adapter achieves the best performance on most datasets.
• Figure 5: The t-SNE Visualization of the Oxford-Pets dataset. Left: Original CLIP Images' Features; Right: CODER.