Gradient-based Sampling for Class Imbalanced Semi-supervised Object Detection

TL;DR

This work comprehensively study the class imbalance problem for SSOD under more challenging scenarios, thus forming the first experimental setting for class imbalanced SSOD (CI-SSOD), and proposes a simple yet effective gradient-based sampling framework that tackles the class imbalance problem from the perspective of two types of confirmation biases.

Abstract

Current semi-supervised object detection (SSOD) algorithms typically assume class balanced datasets (PASCAL VOC etc.) or slightly class imbalanced datasets (MS-COCO, etc). This assumption can be easily violated since real world datasets can be extremely class imbalanced in nature, thus making the performance of semi-supervised object detectors far from satisfactory. Besides, the research for this problem in SSOD is severely under-explored. To bridge this research gap, we comprehensively study the class imbalance problem for SSOD under more challenging scenarios, thus forming the first experimental setting for class imbalanced SSOD (CI-SSOD). Moreover, we propose a simple yet effective gradient-based sampling framework that tackles the class imbalance problem from the perspective of two types of confirmation biases. To tackle confirmation bias towards majority classes, the gradient-based reweighting and gradient-based thresholding modules leverage the gradients from each class to fully balance the influence of the majority and minority classes. To tackle the confirmation bias from incorrect pseudo labels of minority classes, the class-rebalancing sampling module resamples unlabeled data following the guidance of the gradient-based reweighting module. Experiments on three proposed sub-tasks, namely MS-COCO, MS-COCO to Object365 and LVIS, suggest that our method outperforms current class imbalanced object detectors by clear margins, serving as a baseline for future research in CI-SSOD. Code will be available at https://github.com/nightkeepers/CI-SSOD.

Figures (4)

• Figure 1: Differences of our proposed CI-SSOD with respect to other related object detection tasks. "CI-SSOD" represents class imbalanced SSOD. "Majority" and "Minority" represent the majority and minority classes respectively.
• Figure 2: Overview of our proposed gradient-based sampling framework. Our framework consists of three main modules, namely the Gradient-based Reweighting (GbR) module, the Gradient-based Thresholding (GbT) module and the Class-rebalancing Sampling (CrS) module. The GbR module dynamically optimizes a set of graident-based linear equations to obtain the class-wise balancing weights $[w_1, ..., w_{n+1}]$. The GbT module guided by the class-wise balancing weights then performs adaptive thresholding to obtain class balanced pseudo labels from $\boldsymbol{D}_u$. The CrS module performs image-level data resampling on $D_u$ with respective to the confidence and thresholds from GbT. Our method is agnostic to all SSOD methods.
• Figure 3: Comparison of precision and recall of pseudo labels in Soft teacher baseline and our proposed method. Our method generates more pseudo labels for minority classes while preserving the precision of all classes, thus achieving more balanced total detection performance.
• Figure 4: Sample distributions scaled by log10 of all classes in original dataset v.s. our proposed sub-task. (a) COCO v.s. COCO sub-task; (b) Object365 v.s. MS-COCO $\rightarrow$ Object365 sub-task; (c) LVIS v.s. LVIS sub-task. Note that the 'class index' is sorted by class frequency and does not correspond to the actual class number.