Contrastive Credibility Propagation for Reliable Semi-Supervised Learning

TL;DR

This work tackles the reliability gap in semi-supervised learning by introducing Contrastive Credibility Propagation (CCP), a framework that iteratively refines transductive pseudo-labels using credibility vectors to handle label noise and data variability. CCP unifies pseudo-labeling with a generalized, softly supervised contrastive loss and an outer SEAL-inspired loop to mitigate instance-dependent errors, enabling robust performance across few-label, open-set, noisy-label, and class-misalignment scenarios. The approach is complemented by a KL-divergence based subsampling strategy and a structured training pipeline to build a final classifier after iterative refinement. Empirical results on CIFAR-10/100 and text datasets show CCP consistently outperforms or matches a supervised baseline across five realistic data-variable settings, highlighting its practical value for practitioners facing uncertain data quality.

Abstract

Producing labels for unlabeled data is error-prone, making semi-supervised learning (SSL) troublesome. Often, little is known about when and why an algorithm fails to outperform a supervised baseline. Using benchmark datasets, we craft five common real-world SSL data scenarios: few-label, open-set, noisy-label, and class distribution imbalance/misalignment in the labeled and unlabeled sets. We propose a novel algorithm called Contrastive Credibility Propagation (CCP) for deep SSL via iterative transductive pseudo-label refinement. CCP unifies semi-supervised learning and noisy label learning for the goal of reliably outperforming a supervised baseline in any data scenario. Compared to prior methods which focus on a subset of scenarios, CCP uniquely outperforms the supervised baseline in all scenarios, supporting practitioners when the qualities of labeled or unlabeled data are unknown.

Figures (8)

• Figure 1: The credibility vectors for all samples, $\{q_j\}_{j\in I}$, are initialized with the given labels, label guesses, or $\vec{0}$ if unknown. For trusted labeled data, the associated $q_l$ may be permanently clamped to that value. A single CCP iteration first transductively generates new $\{\tilde{q}_j\}_{j\in I}$ for each of $\Xi$ epochs of training via \ref{['q_j']} while minimizing $\mathcal{L}_{\text{SSC}}$. All $\{\tilde{q}_j\}_{j\in I}$ are averaged and scaled such that the maximum of the strongest credibility vector is $1$ to form $\{\hat{q}_j\}_{j\in I}$. We then perform a final credibility adjustment and clip all values outside $[0, 1]$ to form $\{q_j\}_{j\in I}$. Once pseudo-labels have been refined through CCP iterations, they are then used to build an inductive classifier while minimizing both $\mathcal{L}_{\text{SSC}}$ and $\mathcal{L}_{\text{CLS}}$. The (inherently error-prone) results of \ref{['q_j']} are never used to directly supervise error-sensitive contrastive and classifier losses.
• Figure 2: CCP uses an encoder $f_b(\cdot)$ and two projection heads: $f_z(f_b(\cdot))$ for contrastive loss and $f_g(f_b(\cdot))$ for classification loss.
• Figure 3: Top: An unlabeled sample features high similarity scores of 0.99 and 0.98 for both classes. A conventional softmax label is computed as $\hat{y}=[0.502,0.498]$. A credibility vector, $q$, is computed as $0.99-0.98=0.01$ and $0.98-0.99=-0.01$ (clipped to 0 before it's used in Xent). Thus, $q=[0.01,0.0]$. Bottom: Consider $x$ to be the softmax output of a binary classifier for the blue class. Using $\hat{y}$ to compute Xent induces strong gradients at either pole despite the true class being nearly ambiguous. Using $q$ will ensure the gradient for this sample is near $0$ everywhere except $x=0$. If $q=\vec{0}$, i.e. the sample is equally similar to both classes, the gradient is $0$ everywhere.
• Figure 4: Performance of CCP iterations on the base case and few-label experiments of CIFAR-10 with and without credibility. Solid lines indicate the use of credibility adjustments. Dashed lines indicate a softmax function. Top: Pseudo-label accuracy. Bottom: The average strength of correct and incorrect pseudo-labels in the base case.
• Figure 5: Pseudo-label accuracy during CCP iterations in the base case and few-label experiments of CIFAR-10 (top) and CIFAR-100 (bottom). Solid lines depict the usage of $D_{\text{KL}}$-based subsampling with the parameters and schedule presented in \ref{['ccp']} with an initial $d_{\textrm{max}}=0.01$. Dashed lines indicate no subsampling i.e.$d_{\textrm{max}}=0.0$.