Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner

TL;DR

This paper tackles the challenge of adapting large vision–language models without labeled data by introducing Collaborative Fine-Tuning (CoFT). CoFT leverages a dual-model, dual-prompt framework to generate and validate pseudo-labels without hand-crafted thresholds, using high-confidence samples in a two-phase process that first tunes lightweight adapters and then fully fine-tunes the visual encoder. The enhanced variant, CoFT+, adds iterative PEFT, momentum contrastive learning, and LLM-generated prompts to further refine pseudo-labels and representations, achieving state-of-the-art results on nine benchmarks, including real-world noisy data where it outperforms supervised baselines. The work demonstrates that carefully designed cross-modal collaboration and sample-dependent pseudo-label filtering can unlock robust annotation-free adaptation with practical implications for domain-specific tasks and model compression.

Abstract

Large-scale vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization, but adapting them to downstream tasks typically requires costly labeled data. Existing unsupervised self-training methods rely on pseudo-labeling, yet often suffer from unreliable confidence filtering, confirmation bias, and underutilization of low-confidence samples. We propose Collaborative Fine-Tuning (CoFT), an unsupervised adaptation framework that leverages unlabeled data through a dual-model, cross-modal collaboration mechanism. CoFT introduces a dual-prompt learning strategy with positive and negative textual prompts to explicitly model pseudo-label cleanliness in a sample-dependent manner, removing the need for hand-crafted thresholds or noise assumptions. The negative prompt also regularizes lightweight visual adaptation modules, improving robustness under noisy supervision. CoFT employs a two-phase training scheme, transitioning from parameter-efficient fine-tuning on high-confidence samples to full fine-tuning guided by collaboratively filtered pseudo-labels. Building on CoFT, CoFT+ further enhances adaptation via iterative fine-tuning, momentum contrastive learning, and LLM-generated prompts. Extensive experiments demonstrate consistent gains over existing unsupervised methods and even few-shot supervised baselines.

Figures (2)

• Figure 1: Overall framework of the proposed Collaborative Fine-Tuning (CoFT). CoFT adopts a two-phase collaborative training paradigm to adapt CLIP using only unlabeled data. Phase I: Parameter-Efficient Fine-Tuning (PEFT) with high-confidence pseudo-labels. Pseudo-labels are first generated via CLIP zero-shot inference, from which a small set of high-confidence samples is selected. Two CLIP models are then fine-tuned in parallel using lightweight visual adaptation modules and a dual-prompt learning strategy in the textual encoder, where positive and negative prompts explicitly model the cleanliness of pseudo-labels through a dual-loss objective. Phase II: Collaborative pseudo-label filtering and full fine-tuning. The two models perform bidirectional cross-modal collaboration, alternately generating and validating pseudo-labels over the entire unlabeled dataset using the learned positive–negative prompt similarity criterion. High-quality pseudo-labels are retained to fully fine-tune the visual encoder, enabling robust and scalable task-specific adaptation while mitigating confirmation bias.
• Figure 2: Accuracy comparison of different methods incorporating few-shot labeled samples.