When Imbalance Comes Twice: Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation

TL;DR

This work investigates how pool-based active learning for binary semantic segmentation behaves under simulated class imbalance and label shift in industrial defect detection. By constructing controlled datasets with varying $\pi^u$ and $\pi^t$ and comparing Random Sampling, Entropy Sampling, and Core-Set Sampling, the study reveals that entropy- and core-set-based AL generally enhance label efficiency and stability, especially at moderate to high imbalance, though label shift can erode gains. The findings demonstrate AL's practical value for maintaining high-quality defect segmentation with limited annotations and abundant non-defective data, while highlighting robustness limits under deployment-distribution mismatch. The work offers actionable guidance for deploying AL in quality-control pipelines and lays groundwork for further robustness analyses in real-world industrial settings.

Abstract

The aim of Active Learning is to select the most informative samples from an unlabelled set of data. This is useful in cases where the amount of data is large and labelling is expensive, such as in machine vision or medical imaging. Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images. This results, on the one hand, in a strong class imbalance in defect distribution and, on the other hand, in a potential label shift caused by limited storage. To understand how these two forms of imbalance affect active learning algorithms, we propose a simulation study based on two open-source datasets. We artificially create datasets for which we control the levels of class imbalance and label shift. Three standard active learning selection strategies are compared: random sampling, entropy-based selection, and core-set selection. We demonstrate that active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets. We also illustrate and measure the loss of efficiency that occurs in the situation a strong label shift.

Figures (5)

• Figure 1: From top to bottom. (i) Data acquisition: images are collected in the factory according to a production distribution $\pi^t$; (ii) The standard AL setting corresponds to the one where all collected images are stored (situation where $\pi^u=\pi^t$). Due to limited storage capacity, it may happen that only some images are stored according to expert rules (situation $\pi^u>>\pi^t$). (iii) Active Learning procedure: each cycle $j$ selects a pool of images, $S_j^u$, from $D^u$, label selected images (giving rise to $S_j^\ell$), train a model on on $D_j^\ell = D_{j-1}^\ell \cup S_j^\ell$ and uses this trained model $\hat{f}_{\theta,j}$ to select most informative images from $D^u\setminus \cup_{k=0}^j D_k^u$ with an acquisition (or score) function $s$.
• Figure 2: (a) example on an image from the potato disease dataset. (b) True faulty pixels in yellow. (c) Predicted probablities for the defect class. (d) Pixelwise Entropy on the prediction of a trained model. Yellow : Faulty, Purple : Healthy. This is done with a model trained on the full patched dataset for illustration purposes.
• Figure 3: (a) Original Image 512x512, Every image contains defective pixels (b) Patchification into 256x256 patches. This creates two sets of images : the first one composed of the healthy pixels and the second one with the defects only (c) Synthetic Class imbalance by randomly sampling each set of images according to a predefined proportion of defects (d) Test set distributions under AL assumption and industrial case fixed at 5%
• Figure 4: Test F1-scores for the potato diseases dataset, (a), and for the severstal dataset, (b). For each dataset, scores are based on 10 AL cycles from 2% to 20% of the size of the unlabelled dataset. The solid line (resp. bands) represents (resp. represent) the mean curve (resp. interquartile range pointwise bands) obtained over 15 repetitions. The upper row are the results on the test set following the same distribution of the unlabelled dataset (no label shift). The bottom row depicts the performances of the same model but on a test set with a different distribution (where only 5% of images contain faulty pixels).
• Figure 5: (a) Proportion of faulty images selected at each cycle for the potato diseases dataset. (b) Uniqueness scores for the potato disease dataset defined by \ref{['eq:us']}. Remind that a uniqueness score of 0.5 means that among all images selected at cycle $j$, half of them are unique.