A Confidence-Variance Theory for Pseudo-Label Selection in Semi-Supervised Learning

TL;DR

The paper addresses pseudo-label reliability in semi-supervised learning by challenging fixed confidence thresholds, arguing that overconfidence and uneven residual-class distributions undermine accuracy. It derives a Confidence-Variance (CoVar) framework from entropy minimization, decomposing cross-entropy into Maximum Confidence ($p_j(k')$) and Residual-Class Variance ($v_j$) with an adaptive weight $g_j(p_j(k'))$, and introduces a batch-level term involving $ar{MC}$, $sar{RCV}$, and $ ext{Cov}(g,v)$. Pseudo-label selection is reformulated as a spectral relaxation in a confidence-variance feature space, yielding a threshold-free partitioning that can be plugged into segmentation and classification pipelines. Across VOC 2012, Cityscapes, CIFAR-10, and Mini-ImageNet, CoVar delivers consistent gains, especially under low-label and imbalanced conditions, by suppressing overconfident but unstable predictions and improving coverage across classes.

Abstract

Most pseudo-label selection strategies in semi-supervised learning rely on fixed confidence thresholds, implicitly assuming that prediction confidence reliably indicates correctness. In practice, deep networks are often overconfident: high-confidence predictions can still be wrong, while informative low-confidence samples near decision boundaries are discarded. This paper introduces a Confidence-Variance (CoVar) theory framework that provides a principled joint reliability criterion for pseudo-label selection. Starting from the entropy minimization principle, we derive a reliability measure that combines maximum confidence (MC) with residual-class variance (RCV), which characterizes how probability mass is distributed over non-maximum classes. The derivation shows that reliable pseudo-labels should have both high MC and low RCV, and that the influence of RCV increases as confidence grows, thereby correcting overconfident but unstable predictions. From this perspective, we cast pseudo-label selection as a spectral relaxation problem that maximizes separability in a confidence-variance feature space, and design a threshold-free selection mechanism to distinguish high- from low-reliability predictions. We integrate CoVar as a plug-in module into representative semi-supervised semantic segmentation and image classification methods. Across PASCAL VOC 2012, Cityscapes, CIFAR-10, and Mini-ImageNet with varying label ratios and backbones, it consistently improves over strong baselines, indicating that combining confidence with residual-class variance provides a more reliable basis for pseudo-label selection than fixed confidence thresholds. (Code: https://github.com/ljs11528/CoVar_Pseudo_Label_Selection.git)

Figures (8)

• Figure 1: Relationship between pseudo-label accuracy and maximum confidence on PASCAL VOC 2012 (1/4 split). The sample points in this density plot are collected from the training process spanning epoch 1 to 79. The vertical axis represents the mean of the maximum confidence across all pixels, while the horizontal axis is computed using the validation set. Traditional fixed-threshold methods tend to select a number of low-quality pseudo labels while discarding some high-quality ones (mostly from minority classes), which ultimately leads to suboptimal overall pseudo-label accuracy.
• Figure 2: Pseudo-label accuracy and selection ratio under different confidence thresholds on PASCAL VOC 1/4. The blue curve shows the pseudo-label accuracy $a(\tau)=\mathbb{P}(\hat{y}=y \mid \max(p)\ge\tau)$ under a fixed-threshold selection strategy, while the orange curve indicates the proportion of samples whose maximum confidence exceeds $\tau$. The gray dashed line represents the ideal calibration $y=\tau$; the gap at a typical threshold (e.g., $\tau_0=0.95$) visualizes the calibration error $\Delta = \lvert \mathbb{P}(\hat{y}=y \mid \max(p)\ge\tau_0)-\tau_0 \rvert$. The gap further highlights that high-confidence predictions are systematically overconfident, revealing an inherent mismatch between confidence and true accuracy in fixed-threshold methods.
• Figure 3: The overall pipeline of the proposed method. Based on the proposed confidence-variance theory, the prediction separation module divides pseudo labels into high-reliability and low-reliability groups. Then, Gaussian weighting is applied to transform the statistical characteristics of MC and RCV into pseudo-label weights, enabling the selection of high-quality pseudo labels.
• Figure 4: Illustration of the approximate CE loss $\mathrm{CE} = -\log(p_j(k')) + \frac{(K-1)^2}{2(1 - p_j(k'))} \, v_j$ on the plane spanned by model confidence $p_j(k')$ and relative confidence variation $v_j$. The contour map shows that (i) samples with low confidence and large variation (top-left region) incur very high loss, (ii) even high-confidence predictions with large variation (top-right region) are heavily penalized due to the large gradient factor $g_j$, and (iii) the most reliable pseudo-labeled samples concentrate in the high-confidence, low-variation region (bottom-right), where the loss remains minimal.
• Figure 5: Change in pseudo-label selection rate during different model training processes. Experiments are conducted on the blender PASCAL VOC 2012 (321×321) and CIFAR-10. Compared to the fixed threshold method, our approach can significantly improve the pseudo-label selection rate.