Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models

Song-Lin Lv, Rui Zhu, Tong Wei, Yu-Feng Li, Lan-Zhe Guo

TL;DR

This work systematically compares SSL methods with pre-trained vision-language models under a fixed supervision budget to address label scarcity. Across standard, open-set, open-world, and long-tailed settings, pre-trained models (notably CLIP with prompt-tuning) generally offer higher data efficiency and stronger performance than traditional SSL, while SSL struggles with adaptation to distribution shifts and unseen classes. The authors also reveal that very large multimodal language models do not consistently outperform CLIP-based approaches on image classification, highlighting limitations in visual encoders and the need for better vision-focused representations. They conclude with recommendations for integrating SSL and pre-training, and outline directions to advance SSL in the era of large models, including more challenging high-resolution benchmarks and robust adaptation strategies.

Abstract

Semi-supervised learning (SSL) alleviates the cost of data labeling process by exploiting unlabeled data and has achieved promising results. Meanwhile, with the development of large foundation models, exploiting pre-trained models becomes a promising way to address the label scarcity in the downstream tasks, such as various parameter-efficient fine-tuning techniques. This raises a natural yet critical question: When labeled data is limited, should we rely on unlabeled data or pre-trained models? To investigate this issue, we conduct a fair comparison between SSL methods and pre-trained models (e.g., CLIP) on representative image classification tasks under a controlled supervision budget. Experiments reveal that SSL has met its ``Waterloo" in the era of large models, as pre-trained models show both high efficiency and strong performance on widely adopted SSL benchmarks. This underscores the urgent need for SSL researchers to explore new avenues, such as deeper integration between the SSL and pre-trained models. Furthermore, we investigate the potential of Multi-Modal Large Language Models (MLLMs) in image classification tasks. Results show that, despite their massive parameter scales, MLLMs still face significant performance limitations, highlighting that even a seemingly well-studied task remains highly challenging.

Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models

TL;DR

This work systematically compares SSL methods with pre-trained vision-language models under a fixed supervision budget to address label scarcity. Across standard, open-set, open-world, and long-tailed settings, pre-trained models (notably CLIP with prompt-tuning) generally offer higher data efficiency and stronger performance than traditional SSL, while SSL struggles with adaptation to distribution shifts and unseen classes. The authors also reveal that very large multimodal language models do not consistently outperform CLIP-based approaches on image classification, highlighting limitations in visual encoders and the need for better vision-focused representations. They conclude with recommendations for integrating SSL and pre-training, and outline directions to advance SSL in the era of large models, including more challenging high-resolution benchmarks and robust adaptation strategies.

Abstract

Semi-supervised learning (SSL) alleviates the cost of data labeling process by exploiting unlabeled data and has achieved promising results. Meanwhile, with the development of large foundation models, exploiting pre-trained models becomes a promising way to address the label scarcity in the downstream tasks, such as various parameter-efficient fine-tuning techniques. This raises a natural yet critical question: When labeled data is limited, should we rely on unlabeled data or pre-trained models? To investigate this issue, we conduct a fair comparison between SSL methods and pre-trained models (e.g., CLIP) on representative image classification tasks under a controlled supervision budget. Experiments reveal that SSL has met its ``Waterloo" in the era of large models, as pre-trained models show both high efficiency and strong performance on widely adopted SSL benchmarks. This underscores the urgent need for SSL researchers to explore new avenues, such as deeper integration between the SSL and pre-trained models. Furthermore, we investigate the potential of Multi-Modal Large Language Models (MLLMs) in image classification tasks. Results show that, despite their massive parameter scales, MLLMs still face significant performance limitations, highlighting that even a seemingly well-studied task remains highly challenging.
Paper Structure (21 sections, 6 figures, 8 tables)

This paper contains 21 sections, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Settings
  4. Experiments
  5. Baselines Selection
  6. Comparisons in Standard SSL Setting
  7. Comparisons in Open-Set SSL Setting
  8. Comparisons in Open-World SSL Setting
  9. Comparisons in Long-Tailed SSL Setting
  10. Comparisons between VLMs and MLLMs
  11. Conclusions and Discussions
  12. Recommendations for Future Research
  13. Use of Large Language Models (LLMs)
  14. Detailed Experimental Setup
  15. Overal Experimental Setup
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Pipelines of semi-supervised learning and pretrain-finetuning.
  • Figure 2: Performance of different backbones.
  • Figure 3: Performance at different resolutions.
  • Figure 4: Comparison of category embedding alignment in zero-shot and fine-tuned models. The left plot shows the misalignment between fine-grained and coarse-grained category embeddings before fine-tuning, while the right plot illustrates the improved alignment achieved through fine-tuning.
  • Figure 5: Representative error examples where the predicted class differs from the true sample label.
  • ...and 1 more figures