Feature Projection Learning for Better Vision-Language Reasoning

TL;DR

This paper tackles the challenge of adapting Vision-Language Pre-Trained models like CLIP to downstream tasks with limited supervision while keeping training efficiency. It introduces Feature Projection Learning (FPL), which converts classification into a feature projection problem by projecting class prototype features into the query image feature space and reconstructing the query feature map; the reconstruction error serves as the class score and is combined with CLIP's original predictions. The approach yields a closed-form solution for the projection through ridge regression, with learnable regularization $\delta=e^{\mu}$ and a trainable temperature $\epsilon$, plus an orthogonality penalty to encourage discriminative projections. Empirically, FPL achieves state-of-the-art results on 11 few-shot datasets and 4 domain-generalization benchmarks, while maintaining extreme efficiency (e.g., 1 minute of training, ~0.001 GFLOPs, ~0.001M parameters on 16-shot ImageNet), demonstrating strong practical impact for rapid, resource-efficient CLIP adaptation.

Abstract

Vision-Language Pre-Trained models, notably CLIP, that utilize contrastive learning have proven highly adept at extracting generalizable visual features. To inherit the well-learned knowledge of VLP models for downstream tasks, several approaches aim to adapt them efficiently with limited supervision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. In this work, we propose a simple yet efficient and effective method called \textit{\textbf{F}eature \textbf{P}rojection \textbf{L}earning(FPL)} to address these problems. Specifically, we develop a projection model that projects class prototype features into the query image feature space and reconstructs the query image feature map. The negative average squared reconstruction error is used as the class score. In this way, we transform the classification problem into a feature projection problem. The final output of this method is a combination of the prediction from the projection model and the original pre-trained CLIP. Comprehensive empirical evaluations confirm that FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin.

Figures (2)

• Figure 1: Overview of our FPL method. First, we generate the feature map of the query image by $E_{vm}$, for each class, we pool the features generated from the set of few-shot available images by $E_{vm}$ into a unified feature matrix, then we use projector to project these feature matrix into query image feature space and reconstruct the query image feature map. We calculate the negative squared Euclidean distance between the reconstructed query feature map and the real query feature map as the classification score. Combined with the prediction of the original CLIP, we got the final prediction. Note that $E_{vm}$ is the modified image encoder of CLIP that does not include the last attention pooling layer.
• Figure 2: Performance comparisons on few-shot learning on 11 datasets. For each dataset, we report the accuracy on 1-/2-/4-/8-/16-shot settings. The top-left subfigure shows the average accuracy over all 11 datasets.