Mind the Gap: Confidence Discrepancy Can Guide Federated Semi-Supervised Learning Across Pseudo-Mismatch

TL;DR

The paper tackles the challenge of Federated Semi-Supervised Learning under non-IID data, where pseudo-label quality and convergence deteriorate due to data heterogeneity. It introduces SAGE, a framework that leverages confidence discrepancies between local and global models through Collaborative Pseudo-Label Generation (CPG) and Confidence-Driven Soft Correction (CDSC) to produce robust, soft pseudo-labels. Empirical results across CIFAR-10/100, SVHN, and CINIC-10 show that SAGE yields significant performance gains and faster convergence, while also serving as a plug-in to enhance existing FSSL methods. The findings offer a practical approach to robust FSSL in heterogeneous federated environments and point to broader applicability of confidence-driven pseudo-label correction.

Abstract

Federated Semi-Supervised Learning (FSSL) aims to leverage unlabeled data across clients with limited labeled data to train a global model with strong generalization ability. Most FSSL methods rely on consistency regularization with pseudo-labels, converting predictions from local or global models into hard pseudo-labels as supervisory signals. However, we discover that the quality of pseudo-label is largely deteriorated by data heterogeneity, an intrinsic facet of federated learning. In this paper, we study the problem of FSSL in-depth and show that (1) heterogeneity exacerbates pseudo-label mismatches, further degrading model performance and convergence, and (2) local and global models' predictive tendencies diverge as heterogeneity increases. Motivated by these findings, we propose a simple and effective method called Semi-supervised Aggregation for Globally-Enhanced Ensemble (SAGE), that can flexibly correct pseudo-labels based on confidence discrepancies. This strategy effectively mitigates performance degradation caused by incorrect pseudo-labels and enhances consensus between local and global models. Experimental results demonstrate that SAGE outperforms existing FSSL methods in both performance and convergence. Our code is available at https://github.com/Jay-Codeman/SAGE

Figures (16)

• Figure 1: Pseudo-labeling accuracy and test accuracy under varying levels of heterogeneity (smaller $\alpha$ indicates greater heterogeneity). In each communication round, all clients are trained using FedSGD mcmahan2017communication for one local epoch. From (a), we observe that as heterogeneity increases, pseudo-labeling accuracy declines. In (b), the performance gap between SGD-FSSL and Centralized FixMatch indicates the degradation caused by heterogeneity. We observe that when incorrect pseudo-labels are removed, SGD-FSSL can reach the level of centralized performance. In short, (a) and (b) show that data heterogeneity can negatively impact both model convergence and final test performance.
• Figure 2: Differences of the pseudo-labeling ability between local and global models on CIFAR-100. (a) shows the distributions of pseudo-labels with confidence greater than 0.99. As heterogeneity increases (with smaller $\alpha$), the local and global models exhibit opposite trends. The difference is also reflected in the number of pseudo-labels in (b).
• Figure 3: Framework of the proposed SAGE. This framework demonstrates the pseudo-labeling strategy of SAGE in the label-at-all-client scenario. The global model’s pseudo-labels provide supplementary information when the local model lacks confidence and are dynamically adjusted based on confidence discrepancies between the local and global models.
• Figure 4: Convergence curves of SAGE and other baselines on CIFAR-100 with $\alpha=1$.
• Figure 5: Ablation on Dynamic Correction Coefficient $\lambda$.