LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarios

TL;DR

LoFT introduces a parameter-efficient fine-tuning (PEFT) approach on foundation models to tackle long-tailed semi-supervised learning (LTSSL), addressing overconfidence and low-quality pseudo-labels by leveraging a calibrated backbone and logit-adjusted supervision. The method uses a loss $\,\mathcal{L} = \,\mathcal{L}_s + \,\mathcal{L}_u$ with hard/soft pseudo-labeling guided by Maximum Softmax Probability and stabilizes learning across head and tail classes. An open-world extension, LoFT-OW, adds a two-stage out-of-distribution (OOD) filtering mechanism (including a high-threshold zero-shot filter and an MSP-based $c_{ood}$) to improve robustness when unlabeled data include OOD samples. Empirically, LoFT achieves strong results on CIFAR-100-LT and ImageNet-127 with both CLIP and OpenCLIP backbones, often outperforming state-of-the-art LTSSL methods and maintaining effectiveness with limited unlabeled data; LoFT-OW further boosts performance in open-world settings, underscoring its practical impact for real-world imbalanced learning.

Abstract

Long-tailed learning has garnered increasing attention due to its wide applicability in real-world scenarios. Among existing approaches, Long-Tailed Semi-Supervised Learning (LTSSL) has emerged as an effective solution by incorporating a large amount of unlabeled data into the imbalanced labeled dataset. However, most prior LTSSL methods are designed to train models from scratch, which often leads to issues such as overconfidence and low-quality pseudo-labels. To address these challenges, we extend LTSSL into the foundation model fine-tuning paradigm and propose a novel framework: LoFT (Long-tailed semi-supervised learning via parameter-efficient Fine-Tuning). We demonstrate that fine-tuned foundation models can generate more reliable pseudolabels, thereby benefiting imbalanced learning. Furthermore, we explore a more practical setting by investigating semi-supervised learning under open-world conditions, where the unlabeled data may include out-of-distribution (OOD) samples. To handle this problem, we propose LoFT-OW (LoFT under Open-World scenarios) to improve the discriminative ability. Experimental results on multiple benchmarks demonstrate that our method achieves superior performance compared to previous approaches, even when utilizing only 1\% of the unlabeled data compared with previous works.

Figures (6)

• Figure 1: Differences among supervised learning, semi-supervised learning, and semi-supervised learning in open-world scenarios. Pentagrams in yellow and green denote samples of head classes and tail classes, respectively.
• Figure 2: The reliability diagrams on (a) ImageNet-LT and (b) Places365-LT based on training from scratch and PEFT, respectively. The horizontal axis represents confidence, and the vertical axis represents accuracy.
• Figure 3: Illustration of the proposed LoFT-OW. $H(p, q)$ denotes the cross-entropy.
• Figure 4: Visualizations of unlabeled samples and their predicted confidence scores on ImageNet-127. Samples with a green background are assigned reliable pseudo-labels with high confidence, while the sample with a red background is identified as an OOD instance.
• Figure 5: Ablation studies on hyper-parameter $c_u$. The horizontal axis represents the value of $c_u$, and the vertical axis represents the accuracy.