Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

TL;DR

The paper tackles the problem of assessing real-world applicability of deep SSL methods by evaluating them under realistic constraints via a unified reimplementation and evaluation platform, formalizing the data setup with labeled data $\mathcal{D}$ and unlabeled data $\mathcal{D}_{UL}$. It demonstrates that many reported SSL gains diminish when a fixed model and fair tuning budget are used, especially under class-distribution shifts between $\mathcal{D}$ and $\mathcal{D}_{UL}$, and that transfer learning can outperform SSL in some settings. Key findings show that strong fully-supervised baselines with careful regularization can rival SSL, SSL performance is sensitive to unlabeled data quantity and composition, and small validation sets impede reliable comparisons; these results inform when SSL is warranted and how to evaluate it. The authors provide concrete recommendations for evaluation and make their unified implementation publicly available to improve reproducibility and real-world applicability of SSL research.

Abstract

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

Figures (7)

• Figure 1: Behavior of the SSL approaches described in \ref{['sec:methods']} on the "two moons" dataset. We omit "Mean Teacher" and "Temporal Ensembling" (\ref{['sec:temt']}) because they behave like $\Pi$-Model (\ref{['sec:pimodel']}). Each approach was applied to a MLP with three hidden layers, each with 10 ReLU units. When trained on only the labeled data (large black and white dots), the decision boundary (dashed line) does not follow the contours of the data "manifold", as indicated by additional unlabeled data (small grey dots). In a simplified view, the goal of SSL is to leverage the unlabeled data to produce a decision boundary which better reflects the data's underlying structure.
• Figure 2: Test error for each SSL technique on CIFAR-10 (six animal classes) with varying overlap between classes in the labeled and unlabeled data. For example, in "25%", one of the four classes in the unlabeled data is not present in the labeled data. "Supervised" refers to using no unlabeled data. Shaded regions indicate standard deviation over five trials.
• Figure 3: Test error for each SSL technique on SVHN with 1,000 labels and varying amounts of unlabeled images from SVHN-extra. Shaded regions indicate standard deviation over five trials. X-axis is shown on a logarithmic scale.
• Figure 4: Test error for each SSL technique on SVHN and CIFAR-10 as the amount of labeled data varies. Shaded regions indicate standard deviation over five trials. X-axis is shown on a logarithmic scale.
• Figure 5: Average validation error over 10 randomly-sampled nonoverlapping validation sets of varying size. For each SSL approach, we re-evaluated an identical model on each randomly-sampled validation set. The mean and standard deviation of the validation error over the 10 sets are shown as lines and shaded regions respectively. Models were trained on SVHN with 1,000 labels. Validation set sizes are listed relative to the training size (e.g. 10% indicates a size-100 validation set). X-axis is shown on a logarithmic scale.