LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP

TL;DR

LP++ reframes Linear Probe for CLIP as a two-block convex problem that jointly optimizes visual class prototypes and text-embedding blending parameters, yielding $l_{ik} = \bm{f}_i^t (\bm{w}_k + \alpha_k \bm{t}_k)$. A Block Majorize-Minimize descent with data-driven Lipschitz-based step sizes provides efficient, validation-free optimization, delivering strong few-shot CLIP performance while remaining black-box and orders of magnitude faster than many baselines. The approach leverages Lipschitz continuity, closed-form block updates, and an initialization strategy grounded in hard/soft feature means, resulting in robust improvements over standard LP and many adapter/prompt-learning methods across 11 benchmarks. The paper also includes detailed proofs and ablations, showing the benefits of learnable text blending, block-cycling strategies, and training-free variants, positioning LP++ as a strong, practical baseline for future CLIP adaptation work.

Abstract

In a recent, strongly emergent literature on few-shot CLIP adaptation, Linear Probe (LP) has been often reported as a weak baseline. This has motivated intensive research building convoluted prompt learning or feature adaptation strategies. In this work, we propose and examine from convex-optimization perspectives a generalization of the standard LP baseline, in which the linear classifier weights are learnable functions of the text embedding, with class-wise multipliers blending image and text knowledge. As our objective function depends on two types of variables, i.e., the class visual prototypes and the learnable blending parameters, we propose a computationally efficient block coordinate Majorize-Minimize (MM) descent algorithm. In our full-batch MM optimizer, which we coin LP++, step sizes are implicit, unlike standard gradient descent practices where learning rates are intensively searched over validation sets. By examining the mathematical properties of our loss (e.g., Lipschitz gradient continuity), we build majorizing functions yielding data-driven learning rates and derive approximations of the loss's minima, which provide data-informed initialization of the variables. Our image-language objective function, along with these non-trivial optimization insights and ingredients, yields, surprisingly, highly competitive few-shot CLIP performances. Furthermore, LP++ operates in black-box, relaxes intensive validation searches for the optimization hyper-parameters, and runs orders-of-magnitudes faster than state-of-the-art few-shot CLIP adaptation methods. Our code is available at: \url{https://github.com/FereshteShakeri/FewShot-CLIP-Strong-Baseline.git}.

Figures (6)

• Figure 1: Comparison of LP++ with state-of-the-art few-shot CLIP methods in the 1-shot setting across 11 datasets. We compute the mean accuracy and standard deviation using 10 random tasks for each dataset. The error bars indicate the average standard deviation over all 11 datasets. The x-axis represents the run time for one task, averaged over the 11 datasets. Tip-Adapter-F and Tip-Adapter-F$^*$ are two re-implementations of Tip-Adapter-F zhang2022tip, with fixed and grid-search hyper-parameters, respectively (implementation details provided in \ref{['sec:baselines']}).
• Figure 2: Visualization of LP++.
• Figure 3: Quantitative performance of different adaptation methods on the 11 benchmarks (mean), as well as in two other datasets, averaged over 10 tasks (additional figures on the remaining 9 datasets can be found in Appendix, \ref{['sec:additional_results']}).
• Figure 4: Single-block GD performance as a function of different values of the learning rates. The dotted vertical line shows the Lipschitz-based, data-driven learning rate.
• Figure 5: The ranking of different methods using Autorank Herbold2020. The results are averaged over 11 datasets.

Theorems & Definitions (8)

• Lemma 2.1
• Theorem 2.2
• Proposition 1
• proof
• Proposition 2
• proof
• Lemma 5.1
• proof