ULFine: Unbiased Lightweight Fine-tuning for Foundation-Model-Assisted Long-Tailed Semi-Supervised Learning

TL;DR

This work assesses the impact of vision-language foundation models on long-tailed semi-supervised learning and identifies two pitfalls of naive fine-tuning: minority bottleneck and majority overconfidence. It introduces Unbiased Lightweight Fine-tuning (ULFine), comprising Prototype Adaptive Fitting (PAF) and Dual Logit Fusion (DLF) to align and fuse semantic and visual cues, producing unbiased pseudo-labels and balanced classifiers. Across CIFAR10-LT, CIFAR100-LT, STL10-LT, and ImageNet-127, ULFine achieves superior tail performance, reduces false pseudo-labels, and significantly lowers training costs (over tenfold) compared with state-of-the-art LTSSL methods. The results highlight the viability of foundation-model-based LTSSL with targeted, lightweight adaptations that preserve head-class accuracy while markedly improving tail-class generalization. The approach offers practical implications for scalable, data-imbalanced semi-supervised learning in real-world vision tasks.

Abstract

Based on the success of large-scale visual foundation models like CLIP in various downstream tasks, this paper initially attempts to explore their impact on Long-Tailed Semi-Supervised Learning (LTSSL) by employing the foundation model with three strategies: Linear Probing (LP), Lightweight Fine-Tuning (LFT), and Full Fine-Tuning (FFT). Our analysis presents the following insights: i) Compared to LTSSL algorithms trained from scratch, FFT results in a decline in model performance, whereas LP and LFT, although boosting overall model performance, exhibit negligible benefits to tail classes. ii) LP produces numerous false pseudo-labels due to \textit{underlearned} training data, while LFT can reduce the number of these false labels but becomes overconfident about them owing to \textit{biased fitting} training data. This exacerbates the pseudo-labeled and classifier biases inherent in LTSSL, limiting performance improvement in the tail classes. With these insights, we propose a Unbiased Lightweight Fine-tuning strategy, \textbf{ULFine}, which mitigates the overconfidence via confidence-aware adaptive fitting of textual prototypes and counteracts the pseudo-labeled and classifier biases via complementary fusion of dual logits. Extensive experiments demonstrate that ULFine markedly decreases training costs by over ten times and substantially increases prediction accuracies compared to state-of-the-art methods.

Figures (6)

• Figure 1: On the CIFAR100-LT dataset, (a): Comparison of top-1 accuracy of Linear Probing (LP), Lightweight Fine-Tuning (LFT), and Full Fine-Tuning (FFT) with existing LTSSL methods. (b) Comparison of top-1 accuracy of head and tail classes of different strategies. (c) The vertical axes (left and right) indicate the confidence level (area plot) and the sample size (dashed line) of false pseudo-labeling.
• Figure 2: Performance comparison of various methods on the CIFAR10-LT with $N_1$=500, $M_1$=4000, and $\gamma_l$=100. “Consistent”, “Uniform” and “Reversed” correspond to scenarios where the imbalance rate $\gamma_u$ is “100", “1", and “1/100" for the unlabeled dataset, respectively.
• Figure 3: Statistics of relevant results on CIFAR10-LT. Top: Distribution of pseudo-labeled samples obtained using different strategies (bar graph). The dotted lines indicate true distributions. Bottom: The vertical axes (left and right) indicate the confidence level (area plot) and the sample size (dashed line) of false pseudo-labeling.
• Figure 4: Top-1 classification accuracy per class under different distribution settings of CIFAR100-LT, comparing various strategies.
• Figure 5: The vertical axes (left and right) indicate the confidence level (area plot) and the sample size (dashed line) of false pseudo-labeling.

Theorems & Definitions (1)

• Definition 1