Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages

TL;DR

The paper addresses the challenge of adapting large Vision-Language Models with limited labeled data per class (Few-Shot Adaptation) by uncovering a consistent two-phase learning dynamic in PEFT methods. It introduces Two-Stage Few-Shot Adaptation (2SFS), which allocates a fixed compute budget to first learn task-level features via PEFT (notably LayerNorm tuning) and then train a simple linear classifier on top, enabling category-level selective inference at test time. Empirically, 2SFS matches or surpasses state-of-the-art methods across 11 datasets and two task settings (base-to-novel and all-to-all) with fixed hyperparameters, and demonstrates robustness across backbones (ViT-B/16, ViT-B/32, ViT-L/14). The approach avoids reliance on handcrafted templates or external knowledge and provides a simple, hyperparameter-stable recipe for FSA of VLMs, with practical benefits for selective inference. Overall, the work highlights that task-level feature learning can generalize to unseen concepts and that a two-stage strategy yields strong, settings-robust performance.

Abstract

An old-school recipe for training a classifier is to (i) learn a good feature extractor and (ii) optimize a linear layer atop. When only a handful of samples are available per category, as in Few-Shot Adaptation (FSA), data are insufficient to fit a large number of parameters, rendering the above impractical. This is especially true with large pre-trained Vision-Language Models (VLMs), which motivated successful research at the intersection of Parameter-Efficient Fine-tuning (PEFT) and FSA. In this work, we start by analyzing the learning dynamics of PEFT techniques when trained on few-shot data from only a subset of categories, referred to as the ``base'' classes. We show that such dynamics naturally splits into two distinct phases: (i) task-level feature extraction and (ii) specialization to the available concepts. To accommodate this dynamic, we then depart from prompt- or adapter-based methods and tackle FSA differently. Specifically, given a fixed computational budget, we split it to (i) learn a task-specific feature extractor via PEFT and (ii) train a linear classifier on top. We call this scheme Two-Stage Few-Shot Adaptation (2SFS). Differently from established methods, our scheme enables a novel form of selective inference at a category level, i.e., at test time, only novel categories are embedded by the adapted text encoder, while embeddings of base categories are available within the classifier. Results with fixed hyperparameters across two settings, three backbones, and eleven datasets, show that 2SFS matches or surpasses the state-of-the-art, while established methods degrade significantly across settings.

Figures (8)

• Figure 1: We present 2SFS, a technically simple revision of classifier tuning. 2SFS exhibits favorable performance both in all-to-all FSA, where train/test categories coincide, as well as in the more challenging base-to-novel setup, where only a subset of base annotated categories are available, and the test suite further spans a set of unseen (novel) classes. Conversely, setting-specific SOTA approaches zanella2024cliplorayang2024mma degrade between settings.
• Figure 2: The natural emergence of a breakpoint during few-shot adaptation with different PEFT strategies. Before the breakpoint (to the left of the line), PEFT learns good task-level features, showed by joint performance increase on both base and novel categories. After the breakpoint (to the right of the line), PEFT specializes in the available data, incurring unrecoverable performance degradation on novel categories accompanied by consistent improvement in base concepts. Results refer to CLIP radford2021learning with the ViT-B/16 visual backbone dosovitskiy2020image.
• Figure 3: 2SFS. Given a computational budget of $m$ iterations, 2SFS operates in two separate stages. In the First Stage, 2SFS learns task-level features by tuning LayerNorm instances. In the Second Stage, a simple classifier initialized with the text embeddings of base categories learns to separate task-level features. At inference time, 2SFS allows to selectively embed categories. Specifically, only novel categories are embedded by the adapted text encoder, while embeddings of base categories are available as rows within $\Phi_\mathcal{B}$.
• Figure 4: Hyperparameter sweep on $\alpha\in[0.2,0.8]$ with a step size of $0.1$, conducted on ImageNet russakovsky2015imagenet. The optimal value $\alpha=0.6$ is transferred to every other experiment of this work.
• Figure 5: Visualization of the absolute accuracy improvement across the 11 benchmarks in the all-to-all setting, relative to exiting after the first stage, i.e., relative to using $\omega_\mathtt{LN}^*$ only.