Uncertainty-aware Pseudo-label Selection for Positive-Unlabeled Learning

TL;DR

This work tackles imbalanced positive-unlabeled learning by introducing PUUPL, an uncertainty-aware pseudo-labeling framework that uses a deep ensemble to rank unlabeled samples by epistemic uncertainty before pseudo-labeling and, when necessary, reclassifying uncertain pseudo-labels back to unlabeled data. The method combines a PU loss with a learnable loss mix, supports soft pseudo-labels, and employs pseudo-unlabeling to curb confirmation bias, achieving state-of-the-art performance across diverse data modalities (images, text, biological sequences) and a real-world Healthcare dataset. Ablation studies demonstrate that uncertainty quantification substantially improves pseudo-label quality (ECE and NLL) and overall accuracy, while robustness analyses show PUUPL tolerates prior misspecification and works with varying numbers of labeled positives and different loss functions. Collectively, PUUPL offers a practical, data-modality-agnostic enhancement for PU learning in imbalanced settings, with significant implications for real-world applications where labeled negatives are scarce or costly.

Abstract

Positive-unlabeled learning (PUL) aims at learning a binary classifier from only positive and unlabeled training data. Even though real-world applications often involve imbalanced datasets where the majority of examples belong to one class, most contemporary approaches to PUL do not investigate performance in this setting, thus severely limiting their applicability in practice. In this work, we thus propose to tackle the issues of imbalanced datasets and model calibration in a PUL setting through an uncertainty-aware pseudo-labeling procedure (PUUPL): by boosting the signal from the minority class, pseudo-labeling expands the labeled dataset with new samples from the unlabeled set, while explicit uncertainty quantification prevents the emergence of harmful confirmation bias leading to increased predictive performance. Within a series of experiments, PUUPL yields substantial performance gains in highly imbalanced settings while also showing strong performance in balanced PU scenarios across recent baselines. We furthermore provide ablations and sensitivity analyses to shed light on PUUPL's several ingredients. Finally, a real-world application with an imbalanced dataset confirms the advantage of our approach.

Figures (5)

• Figure 1: PUUPL is a pseudo-labeling framework for PU learning that uses the epistemic uncertainty of an ensemble to select confident examples to pseudo-label. The ensemble can be trained with any PU loss for PU data while minimizing the cross-entropy loss on the previously assigned pseudo-labels. In a toy example, a single network is not very confident on most of the unlabeled data (a), resulting in many high-confidence incorrect predictions and many low-confidence correct ones (c). The epistemic uncertainty of an ensemble is, on the other hand, very low on most of the unlabeled data (b), resulting in most correct predictions having low uncertainty and most incorrect predictions having high uncertainty (d). Thus, the estimated uncertainty by ensemble can be used more reliably to rank predictions and select correct ones (e). Re-training the model with an increased number of labeled samples will result in a slightly more accurate model, than can be used to predict new pseudo-labels, which will further improve the model's performance, etc.
• Figure 2: Validation accuracy (left, blue) and expected calibration error (ECE, right, green) for a run on CIFAR-10 with 1,000 positives. Note the substantial reduction in ECE in the second and third pseudo-labeling iterations, when the ensemble is trained on soft labels. The orange line indicates the best validation accuracy at each epoch, with the new highest accuracy marked by orange dots. The overall highest was 90.76% at epoch 1092, corresponding to a test accuracy of 90.35%.
• Figure 3: Mean and standard deviation of the CIFAR-10 test accuracy obtained over five runs when training with wrong prior (a), number of training labeled positives (b) and different loss combination parameter $\lambda$ (c). PUUPL proved to be more robust to prior misspecification (true $\pi=0.4$), as the performance degradation was considerably reduced over a wide range of values. It was also more robust to the lower number of labeled samples, as the gap between our framework and nnPU widened when fewer labeled positives were available for training (note the different $y$-axes scales).
• Figure 4: Predicting the outcome of each event in the antigen processing pathway antigen_pathway is crucial to enable the design of epitope vaccines. Vaccines ingested by antigen presenting cells (1a) as well as mutated proteins produced by cancerous cells (1b) are cleaved in short fragments by the proteasome (2). Some of these fragments, or peptides, are then transported into the endoplasmic reticulum (ER) through the Transporter associated with Antigen Processing (TAP). A fraction of these peptides bind to the Major Histocompatibility Complex (MHC, 3) and the resulting construct is then expressed on the cell surface (4), where they can be inspected by passerby T-cells and possibly trigger an appropriate immune response (5).
• Figure 5: Mean and standard deviation of the test accuracy obtained over five runs by different variations of our PUUPL algorithm: (a) different weight initialization at each iteration, (b) balanced or imbalanced PL selection, (c) type of uncertainty, (d) whether to use PN or PU validation set. Note the different scales on the $y$-axes.