SeMi: When Imbalanced Semi-Supervised Learning Meets Mining Hard Examples

TL;DR

This work tackles class-imbalance in semi-supervised learning by exploiting hard unlabeled examples through SeMi, a FixMatch-based framework. It combines Online Hard Example Mining and Learning (OHEML), Pseudo-Label Certainty Enhancement (PLCE), and a Balanced Confidence Decay Memory Bank with prototype-based semantic labeling, along with decoupled learning via Logit Align to reduce head-class bias. The approach yields state-of-the-art CISSL performance across CIFAR-10/100-LT, STL10-LT, and ImageNet-127, with especially large gains in reversed and tail-heavy distributions. The method remains simple to integrate with existing SSL pipelines and has practical impact for real-world data imbalance scenarios.

Abstract

Semi-Supervised Learning (SSL) can leverage abundant unlabeled data to boost model performance. However, the class-imbalanced data distribution in real-world scenarios poses great challenges to SSL, resulting in performance degradation. Existing class-imbalanced semi-supervised learning (CISSL) methods mainly focus on rebalancing datasets but ignore the potential of using hard examples to enhance performance, making it difficult to fully harness the power of unlabeled data even with sophisticated algorithms. To address this issue, we propose a method that enhances the performance of Imbalanced Semi-Supervised Learning by Mining Hard Examples (SeMi). This method distinguishes the entropy differences among logits of hard and easy examples, thereby identifying hard examples and increasing the utility of unlabeled data, better addressing the imbalance problem in CISSL. In addition, we maintain a class-balanced memory bank with confidence decay for storing high-confidence embeddings to enhance the pseudo-labels' reliability. Although our method is simple, it is effective and seamlessly integrates with existing approaches. We perform comprehensive experiments on standard CISSL benchmarks and experimentally demonstrate that our proposed SeMi outperforms existing state-of-the-art methods on multiple benchmarks, especially in reversed scenarios, where our best result shows approximately a 54.8\% improvement over the baseline methods.

Figures (6)

• Figure 1: Differences between previous CISSL methods and our method (SeMi) in generating pseudo-labels. The previous method used a high threshold, missing opportunities to learn from hard examples, possibly from the tail. Our method effectively uses hard examples with techniques like online hard examples mining and learning (OHEML) and pseudo-label certainty enhancement.
• Figure 2: The pipeline of the SeMi framework. The unlabeled data is transformed into weak and strong views by image augmentation. In the strong views branch, the generated features are pushed into the cells only when their confidences reach the Balanced Confidence Decay Memory Bank's ($\mathcal{B}$) threshold. Then, the prototype centers for each category can be calculated, and semantic pseudo-labels for query embeddings are obtained. Meanwhile, the linear pseudo-labels generated from the weak views branch are mixed with the semantic pseudo-labels to get the certainty-enhanced pseudo-labels. Online hard example mining and learning to discriminate the hardness of strong views and give larger weights to the hard examples. For ultra-hard samples, the Embedding Align approach can accelerate the learning. In addition, a balanced classifier is maintained to yield predictions that are more friendly to the tail classes.
• Figure 3: Confusion matrix comparison of multiple CISSL methods.
• Figure 4: Comparison among various CISSL methods with t-SNE visualization.
• Figure 5: (a) Unlabeled and labeled data distribution. (b) Per-class accuracy: DASO vs. our method on a balanced test set.