Impact of Strategic Sampling and Supervision Policies on Semi-supervised Learning

TL;DR

This work explores a number of unsupervised methods for selecting specific subsets of data to label (without prior knowledge of their labels) with the goal of maximizing representativeness w.r.t. the unlabelled set.

Abstract

In semi-supervised representation learning frameworks, when the number of labelled data is very scarce, the quality and representativeness of these samples become increasingly important. Existing literature on semi-supervised learning randomly sample a limited number of data points for labelling. All these labelled samples are then used along with the unlabelled data throughout the training process. In this work, we ask two important questions in this context: (1) does it matter which samples are selected for labelling? (2) does it matter how the labelled samples are used throughout the training process along with the unlabelled data? To answer the first question, we explore a number of unsupervised methods for selecting specific subsets of data to label (without prior knowledge of their labels), with the goal of maximizing representativeness w.r.t. the unlabelled set. Then, for our second line of inquiry, we define a variety of different label injection strategies in the training process. Extensive experiments on four popular datasets, CIFAR-10, CIFAR-100, SVHN, and STL-10, show that unsupervised selection of samples that are more representative of the entire data improves performance by up to ~2% over the existing semi-supervised frameworks such as MixMatch, ReMixMatch, FixMatch and others with random sample labelling. We show that this boost could even increase to 7.5% for very few-labelled scenarios. However, our study shows that gradually injecting the labels throughout the training procedure does not impact the performance considerably versus when all the existing labels are used throughout the entire training.

Figures (9)

• Figure 1: Visual illustration of strategic sampling for a labelling budget of 6 samples. We first cluster the embedding space into 6 clusters (red dots represent the cluster centres) and then select the sample closest to each centroid for labelling.
• Figure 2: Illustrations of explored supervision policies. Here, (a) is the naive supervision policy, and (b) to (f) are explored policies.
• Figure 3: A t-SNE visualization of the unlabelled and selected samples shows that the selected samples are distributed across the entire embedding space. Samples that are in close proximity to the centroids exhibit strikingly similar appearances to the centroid and are likely to contain redundant information.
• Figure 4: Cluster purity for extracted features of different pre-trained encoders on CIFAR-10 dataset.
• Figure 5: The trend in accuracy for (a) random vs. strategic sampling, and (b) imbalanced vs. balanced sampling, for different numbers of samples per class.