Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models

Rui Zhu, Song-Lin Lv, Zi-Kang Wang, Lan-Zhe Guo

TL;DR

Bi-CoG tackles label-scarce vision-language fine-tuning by coupling pre-trained VLMs with semi-supervised learning through a three-stage pseudo-labeling pipeline. It combines inter-model majority voting, intra-model view-based consistency, and an error-aware dynamic budget (guided by Lemma 1 and Theorem 1) to generate high-quality, low-bias pseudo-labels. Theoretical analyses are provided, and empirical results across 14 datasets show consistent gains over strong baselines in both open-world and SSL settings, with up to about 5.69 percentage-point improvements in HM. The approach is plug-and-play, reducing hyperparameter sensitivity and offering robust improvements for real-world, label-scarce vision-language tasks.

Abstract

Exploiting unlabeled data through semi-supervised learning (SSL) or leveraging pre-trained models via fine-tuning are two prevailing paradigms for addressing label-scarce scenarios. Recently, growing attention has been given to combining fine-tuning of pre-trained vision-language models (VLMs) with SSL, forming the emerging paradigm of semi-supervised fine-tuning. However, existing methods often suffer from model bias and hyperparameter sensitivity, due to reliance on prediction consistency or pre-defined confidence thresholds. To address these limitations, we propose a simple yet effective plug-and-play methodology named $\underline{\textbf{Bi-Co}}$nsistency-$\underline{\textbf{G}}$uided Self-Training (Bi-CoG), which assigns high-quality and low-bias pseudo-labels, by simultaneously exploiting inter-model and intra-model consistency, along with an error-aware dynamic pseudo-label assignment strategy. Both theoretical analysis and extensive experiments over 14 datasets demonstrate the effectiveness of Bi-CoG, which consistently and significantly improves the performance of existing methods.

Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models

TL;DR

Bi-CoG tackles label-scarce vision-language fine-tuning by coupling pre-trained VLMs with semi-supervised learning through a three-stage pseudo-labeling pipeline. It combines inter-model majority voting, intra-model view-based consistency, and an error-aware dynamic budget (guided by Lemma 1 and Theorem 1) to generate high-quality, low-bias pseudo-labels. Theoretical analyses are provided, and empirical results across 14 datasets show consistent gains over strong baselines in both open-world and SSL settings, with up to about 5.69 percentage-point improvements in HM. The approach is plug-and-play, reducing hyperparameter sensitivity and offering robust improvements for real-world, label-scarce vision-language tasks.

Abstract

Exploiting unlabeled data through semi-supervised learning (SSL) or leveraging pre-trained models via fine-tuning are two prevailing paradigms for addressing label-scarce scenarios. Recently, growing attention has been given to combining fine-tuning of pre-trained vision-language models (VLMs) with SSL, forming the emerging paradigm of semi-supervised fine-tuning. However, existing methods often suffer from model bias and hyperparameter sensitivity, due to reliance on prediction consistency or pre-defined confidence thresholds. To address these limitations, we propose a simple yet effective plug-and-play methodology named nsistency-uided Self-Training (Bi-CoG), which assigns high-quality and low-bias pseudo-labels, by simultaneously exploiting inter-model and intra-model consistency, along with an error-aware dynamic pseudo-label assignment strategy. Both theoretical analysis and extensive experiments over 14 datasets demonstrate the effectiveness of Bi-CoG, which consistently and significantly improves the performance of existing methods.
Paper Structure (31 sections, 24 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 31 sections, 24 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-efficient Fine-tuning on VLMs
  4. Pseudo-labeling in Semi-supervised Learning
  5. Preliminary
  6. CLIP.
  7. Prompt Tuning.
  8. Problem Definition.
  9. Methodology
  10. Inter-model Consistency
  11. Intra-model Consistency
  12. Error-aware Filtering
  13. Lemma 1.
  14. Theorem 1.
  15. Experiment
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Comparison of existing pseudo-labeling paradigms and our proposed Bi-CoG. (a) Majority voting from multiple VLMs; (b) Top-$k$ selection based on predicted logits; (c) Bi-CoG dynamically generate accurate and unbiased pseudo-labels based on bi-consistency and model error.
  • Figure 2: The overall framework of Bi-CoG's pseudo-label selection. During the self-training phase, Bi-CoG employs inter-model consistency, intra-model consistency, and error-aware filtering to generate accurate and unbiased pseudo-labels, which are then used to fine-tune the target VLM.
  • Figure 3: (a) Pseudo-label accuracy over training iterations on the DTD dataset under different pseudo-labeling strategies; (b) Distribution of pseudo-labels on the EuroSAT dataset before and after applying the intra-model consistency.
  • Figure 4: Comparison between the estimated and true error rate ratios on StanfordCars under the open-world setting. The scaling factor is fixed at $\alpha = 1$ throughout training.

Theorems & Definitions (2)

  • proof
  • proof