Improving realistic semi-supervised learning with doubly robust estimation
Khiem Pham, Charles Herrmann, Ramin Zabih
TL;DR
The paper tackles realistic long-tailed semi-supervised learning (RTSSL) where the unlabeled class distribution $P(Y|A=0)$ is unknown and differs from the labeled distribution. It introduces a two-stage approach: first estimate $P(Y|A=0)$ using a doubly robust estimator grounded in label-shift EM, then plug this estimate into existing pseudo-labeling methods to train the final classifier. The authors prove that the doubly robust estimator achieves asymptotic efficiency under mild rates and demonstrate substantial empirical gains across CIFAR-10/LT, CIFAR-100/LT, STL-10, and Imagenet-127 under varied unlabeled distributions. The method shows robustness to imperfect first-stage estimates and integrates smoothly with established SSL frameworks like SimPro and FixMatch, offering a practical path for realistic SSL in the presence of label shift.
Abstract
A major challenge in Semi-Supervised Learning (SSL) is the limited information available about the class distribution in the unlabeled data. In many real-world applications this arises from the prevalence of long-tailed distributions, where the standard pseudo-label approach to SSL is biased towards the labeled class distribution and thus performs poorly on unlabeled data. Existing methods typically assume that the unlabeled class distribution is either known a priori, which is unrealistic in most situations, or estimate it on-the-fly using the pseudo-labels themselves. We propose to explicitly estimate the unlabeled class distribution, which is a finite-dimensional parameter, \emph{as an initial step}, using a doubly robust estimator with a strong theoretical guarantee; this estimate can then be integrated into existing methods to pseudo-label the unlabeled data during training more accurately. Experimental results demonstrate that incorporating our techniques into common pseudo-labeling approaches improves their performance.