Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model

TL;DR

This paper tackles zero-/few-shot adaptation of vision-language models without task-specific tuning by introducing ECALP, a training-free, graph-based framework that constructs a graph from text prompts, few-shot samples, and test samples and performs label propagation. It tackles three core challenges: (i) efficient, inductive inference via dynamic graph expansion; (ii) leveraging relevant task information through context-aware edge re-weighting; and (iii) a unified, iterative propagation scheme that avoids costly closed-form solutions. The method yields state-of-the-art results on fine-grained classification and distribution-shift tasks, demonstrates robustness across multiple VLM backbones, and maintains fixed hyperparameters, offering practical, real-time applicability. The approach significantly speeds up inference compared with training-based adaptation while preserving or improving accuracy, highlighting a scalable direction for deploying VLMs in diverse downstream settings, with a theoretical basis in label propagation and graph-based manifold learning. The key mathematical components include iterative label propagation with $Y^{t+1}=\alpha\\tilde{W}Y^t+(1-\\alpha)Y^0$ and convergence to $Y^\infty=(I-\\alpha\\tilde{W})^{-1}Y^0$, as well as dynamic graph expansion reducing complexity to $O(d\\,N_v^2)$ and context-aware edge weighting using statistics from text and few-shot distributions.

Abstract

Vision-language models (VLMs) have revolutionized machine learning by leveraging large pre-trained models to tackle various downstream tasks. Although label, training, and data efficiency have improved, many state-of-the-art VLMs still require task-specific hyperparameter tuning and fail to fully exploit test samples. To overcome these challenges, we propose a graph-based approach for label-efficient adaptation and inference. Our method dynamically constructs a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specific tuning. Unlike existing zero-shot label propagation techniques, our approach requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion. We further introduce a context-aware feature re-weighting mechanism to improve task adaptation accuracy. Additionally, our method supports efficient graph expansion, enabling real-time inductive inference. Extensive evaluations on downstream tasks, such as fine-grained categorization and out-of-distribution generalization, demonstrate the effectiveness of our approach. The source code is available at https://github.com/Yushu-Li/ECALP.

Figures (9)

• Figure 1: Illustration of the overall framework of ECALP. We use the text prompts, few-shot samples, and test samples to build the graph. The graph is dynamically expanded upon seeing a new sample with context-aware re-weighted edge weights. An iterative solution is adopted to predict labels for all test samples.
• Figure 2: Few-shot adaptation of VLM for fine-grained categorization downstream tasks with CLIP-RN50 model.
• Figure 3: Wall-clock time comparison between statically-constructed graphs and dynamically-expansive graphs.
• Figure 4: Combining our proposed ECALP with CoOp zhou2022learning on the DTD dataset.
• Figure 5: Effect of label propagation iterations on the DTD dataset.