Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation

Shuvendu Roy, Ali Etemad

TL;DR

A weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance, and a weakly-supervised sampling technique that selects a diverse and representative labelled set that can be seamlessly integrated into existing methods to enhance their performance.

Abstract

We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.

SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation

TL;DR

A weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance, and a weakly-supervised sampling technique that selects a diverse and representative labelled set that can be seamlessly integrated into existing methods to enhance their performance.

Abstract

We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.
Paper Structure (18 sections, 6 equations, 4 figures, 9 tables)

This paper contains 18 sections, 6 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Preliminaries and Problem formulation
  5. SelfPrompt
  6. Weakly-supervised sampling
  7. Experiments
  8. Experiment Setup
  9. Standard Semi-supervised learning
  10. Active Semi-supervised Learning
  11. Base-to-novel generalization
  12. Ablation
  13. Computational complexity
  14. Conclusion
  15. Appendix
  16. ...and 3 more sections

Figures (4)

  • Figure 1: (left) Pseudo-label accuracy; (right) Test accuracy over training sessions.
  • Figure 2: Performance comparison to prior works on semi-supervised tuning of VLMs.
  • Figure 3: (left) A visual illustration of the weakly-supervised sampling module. Using predictions from the pre-trained VLM, the least and most confident samples, which are not representative of the downstream data, are filtered out. The remaining feature space is then clustered into a number of clusters equal to the labelling budget to ensure maximum diversity among the selected samples. (right) Cluster-guided pseudo-labelling assigns the same class label to samples near the cluster centers as the pseudo-label.
  • Figure 4: Qualitative analysis of weakly-supervised sampling and cluster-guided pseudo-labelling with two classes (fist and cat).