Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model

TL;DR

This work challenges the conventional view that unseen-class unlabeled data inevitably harms semi-supervised learning (SSL) performance. It introduces RE-SSL, a causal-model–driven evaluation framework that decouples seen and unseen unlabeled data and varies five critical factors to study global and local robustness of SSL methods. By redefining dataset construction (fixing $r_s$ while varying $r_u$) and proposing a suite of metrics including $R_{slope}$, GM, WAD, BAD, and $P_{AD\ge 0}$, the paper provides a rigorous, multi-faceted assessment across diverse datasets (CIFAR-10/100, Imagenet-100, and others) and algorithms. Key findings show that unseen-class unlabeled data does not inevitably degrade SSL performance; several methods (e.g., PseudoLabel, ICT, UASD, CAFA) exhibit robustness, while some FixMatch-based approaches remain sensitive under certain conditions. The results highlight nuanced dependencies on unseen-class quantity, category diversity, and distributional nearness, offering practical guidance for deploying SSL in open or evolving class settings and advancing robust SSL design.

Abstract

Semi-supervised learning (SSL) effectively leverages unlabeled data and has been proven successful across various fields. Current safe SSL methods believe that unseen classes in unlabeled data harm the performance of SSL models. However, previous methods for assessing the impact of unseen classes on SSL model performance are flawed. They fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning the decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather a decrease in seen class samples. Thus, the prior flawed assessment standard that ``unseen classes in unlabeled data can damage SSL model performance" may not always hold true. This paper strictly adheres to the principle of controlling variables, maintaining the proportion of seen classes in unlabeled data while only changing the unseen classes across five critical dimensions, to investigate their impact on SSL models from global robustness and local robustness. Experiments demonstrate that unseen classes in unlabeled data do not necessarily impair the performance of SSL models; in fact, under certain conditions, unseen classes may even enhance them.

Figures (3)

• Figure 1: The structural causal model of unseen-class evaluation process. $D_L, D_U^S, D_U^U$, and $Acc$ denote labeled data, seen-class unlabeled data, unseen-class unlabeled data, and SSL model's performance of seen-class classification, respectively. The dashed line represents a confounding factor, which is the fundamental reason for the failure of previous evaluations.
• Figure 2: An example of dataset construction in the previous evaluation and our RE-SSL evaluation framework. $r_s$ denotes the ratio of selected seen-class data to all seen-class data. $r_u$ denotes the ratio of selected unseen-class data to all unseen-class data. $P$ denotes the distribution of seen classes, and $Q$ denotes the distribution of unseen classes. $D_B^S$ and $D_B^U$ denote the initial set of seen classes and unseen classes, respectively. $D_L$ denotes the labeled set, $D_U$ denotes the unlabeled set. $D_U^S$ and $D_U^U$ together form $D_U$, where $D_U^S$ and $D_U^U$ are sampled from $D_B^S$ and $D_B^U$ according to $r_s$ and $r_u$, respectively. The red bidirectional arrow represents the confounding factor mentioned in Figure 1.
• Figure 3: GM metrics on CIFAR10 with 100 labels under different factors.

Theorems & Definitions (2)

• Definition 1
• Definition 2