Quantized Prompt for Efficient Generalization of Vision-Language Models

TL;DR

This work addresses the challenge of adapting large vision–language models to downstream tasks without succumbing to overfitting or catastrophic forgetting. It introduces QPrompt, a quantization-based regularization strategy that treats quantization error as a controllable noise source, implemented via K-Means clustering and a constrained adaptive clustering schedule to quantize prompts and select weights with minimal performance loss. The approach yields state-of-the-art or competitive results on base-to-new generalization, domain generalization, cross-dataset transfer, and few-shot learning while dramatically reducing storage requirements (e.g., enabling sub-kilobyte prompt representations). The findings demonstrate that moderate quantization can enhance generalization, enable efficient deployment on resource-limited devices, and be integrated with existing prompt-tuning methods like CoOp and MaPLe, broadening the practical impact for vision–language adaptation.

Abstract

In the past few years, large-scale pre-trained vision-language models like CLIP have achieved tremendous success in various fields. Naturally, how to transfer the rich knowledge in such huge pre-trained models to downstream tasks and datasets becomes a hot topic. During downstream adaptation, the most challenging problems are overfitting and catastrophic forgetting, which can cause the model to overly focus on the current data and lose more crucial domain-general knowledge. Existing works use classic regularization techniques to solve the problems. As solutions become increasingly complex, the ever-growing storage and inference costs are also a significant problem that urgently needs to be addressed. While in this paper, we start from an observation that proper random noise can suppress overfitting and catastrophic forgetting. Then we regard quantization error as a kind of noise, and explore quantization for regularizing vision-language model, which is quite efficiency and effective. Furthermore, to improve the model's generalization capability while maintaining its specialization capacity at minimal cost, we deeply analyze the characteristics of the weight distribution in prompts, conclude several principles for quantization module design and follow such principles to create several competitive baselines. The proposed method is significantly efficient due to its inherent lightweight nature, making it possible to adapt on extremely resource-limited devices. Our method can be fruitfully integrated into many existing approaches like MaPLe, enhancing accuracy while reducing storage overhead, making it more powerful yet versatile. Extensive experiments on 11 datasets shows great superiority of our method sufficiently. Code is available at https://github.com/beyondhtx/QPrompt.

Figures (7)

• Figure 1: Overall performance comparison with existing vision-language tuning methods. Our method outperforms all of the state-of-the-art competitors with significantly fewer storage space. Based on the proposed quantization algorithm, our method could be integrated into many of the existing methods and bring consistent improvements with excellent efficiency.
• Figure 2: The variations of the specialization capability represented by the test accuracy on base, seen classes and the generalization capability represented by the test accuracy on new, unseen classes on average of the ten datasets in base-to-new generalization setting in \ref{['sec:experiments']} except for time-consuming ImageNet. The curves of different colors represent the data under the influence of random Gaussian noise of different intensities, e.g."Noise_0.01" adds random noise with a distribution of $\mathcal{N}(0, 0.01^{2})$ to the prompt. "Noise_0" denotes the baseline prompt tuning. As training progresses, the generalization capability of baseline prompt tuning continuously decreases while the specialization capability improves. Therefore, we expect that adding noise can achieve a better balance between generalization and specialization. However, excessive noise, e.g.0.1, greatly diminishes the model's specialization capability, while insufficient noise, e.g.0.001, fails to provide effective regularization. Only noise of moderate intensity outperforms baseline in specialization-generalization trade-off, effectively enhancing the unseen class accuracy without significantly compromising seen class accuracy.
• Figure 3: The histogram about the frequency of weights of prompts during training of CoOp zhou2021coop on eurosat dataset. We find the shape of the prompt's weight distribution remains largely unchanged throughout the entire training, but the variance of the weight distribution increases rapidly at the beginning of training. Additionally, we also notice that there are almost no outliers in the prompt's weights throughout the entire training process, and apart from the unstable initial training phase, the changes in weights between adjacent phases are not significant, indicating a very gentle overall updating trend.
• Figure 4: Overview of Quantized Prompt. We set $b=2$ for a clear explanation. Normalization and denormalization processes are not shown here.
• Figure 5: Average few-shot learning results on 11 datasets.