Soft Label PU Learning

TL;DR

This work introduces soft label PU learning, where unlabeled samples receive soft labels reflecting their likelihood of being positive. It defines PU substitute metrics $TPR_{SPU}$, $FPR_{SPU}$, and $AUC_{SPU}$ and analyzes their relationship to real metrics under Generalized SCAR, Monotonic, and Noisy Monotonic assumptions, showing that improvements in PU metrics often translate to improvements in true metrics. A simple parametric learning objective is proposed to optimize these PU metrics, with theoretical guarantees and convergence results. Empirical validation on UCI, image datasets, and Tencent's anti-cheat task demonstrates consistent gains over baseline PU methods, supporting practical utility in scenarios with uneven labeling and rich domain knowledge.

Abstract

PU learning refers to the classification problem in which only part of positive samples are labeled. Existing PU learning methods treat unlabeled samples equally. However, in many real tasks, from common sense or domain knowledge, some unlabeled samples are more likely to be positive than others. In this paper, we propose soft label PU learning, in which unlabeled data are assigned soft labels according to their probabilities of being positive. Considering that the ground truth of TPR, FPR, and AUC are unknown, we then design PU counterparts of these metrics to evaluate the performances of soft label PU learning methods within validation data. We show that these new designed PU metrics are good substitutes for the real metrics. After that, a method that optimizes such metrics is proposed. Experiments on public datasets and real datasets for anti-cheat services from Tencent games demonstrate the effectiveness of our proposed method.

Figures (2)

• Figure 1: Illustration of the error of the ROC curve obtained by optimizing PU metrics.
• Figure 2: Procedure of game-cheater check and punishment.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Lemma 1