Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregation

Marawan Elbatel, Hualiang Wang, Jixiang Chen, Hao Wang, Xiaomeng Li

TL;DR

This paper enables unlabeled client aggregation through SemiAnAgg, a novel Semi-supervised Anchor-Based federated Aggregation, effectively harnessing their informative value by feeding local client data to the same global model and the same consistently initialized anchor model.

Abstract

Federated semi-supervised learning (FedSemi) refers to scenarios where there may be clients with fully labeled data, clients with partially labeled, and even fully unlabeled clients while preserving data privacy. However, challenges arise from client drift due to undefined heterogeneous class distributions and erroneous pseudo-labels. Existing FedSemi methods typically fail to aggregate models from unlabeled clients due to their inherent unreliability, thus overlooking unique information from their heterogeneous data distribution, leading to sub-optimal results. In this paper, we enable unlabeled client aggregation through SemiAnAgg, a novel Semi-supervised Anchor-Based federated Aggregation. SemiAnAgg learns unlabeled client contributions via an anchor model, effectively harnessing their informative value. Our key idea is that by feeding local client data to the same global model and the same consistently initialized anchor model (i.e., random model), we can measure the importance of each unlabeled client accordingly. Extensive experiments demonstrate that SemiAnAgg achieves new state-of-the-art results on four widely used FedSemi benchmarks, leading to substantial performance improvements: a 9% increase in accuracy on CIFAR-100 and a 7.6% improvement in recall on the medical dataset ISIC-18, compared with prior state-of-the-art. Code is available at: https://github.com/xmed-lab/SemiAnAgg.

Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregation

TL;DR

This paper enables unlabeled client aggregation through SemiAnAgg, a novel Semi-supervised Anchor-Based federated Aggregation, effectively harnessing their informative value by feeding local client data to the same global model and the same consistently initialized anchor model.

Abstract

Federated semi-supervised learning (FedSemi) refers to scenarios where there may be clients with fully labeled data, clients with partially labeled, and even fully unlabeled clients while preserving data privacy. However, challenges arise from client drift due to undefined heterogeneous class distributions and erroneous pseudo-labels. Existing FedSemi methods typically fail to aggregate models from unlabeled clients due to their inherent unreliability, thus overlooking unique information from their heterogeneous data distribution, leading to sub-optimal results. In this paper, we enable unlabeled client aggregation through SemiAnAgg, a novel Semi-supervised Anchor-Based federated Aggregation. SemiAnAgg learns unlabeled client contributions via an anchor model, effectively harnessing their informative value. Our key idea is that by feeding local client data to the same global model and the same consistently initialized anchor model (i.e., random model), we can measure the importance of each unlabeled client accordingly. Extensive experiments demonstrate that SemiAnAgg achieves new state-of-the-art results on four widely used FedSemi benchmarks, leading to substantial performance improvements: a 9% increase in accuracy on CIFAR-100 and a 7.6% improvement in recall on the medical dataset ISIC-18, compared with prior state-of-the-art. Code is available at: https://github.com/xmed-lab/SemiAnAgg.
Paper Structure (24 sections, 4 equations, 12 figures, 9 tables)

This paper contains 24 sections, 4 equations, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semi-Supervised Learning
  4. Federated Semi-Supervised Learning
  5. Federated Model Aggregation
  6. Preliminaries on Federated Semi-Supervised Learning
  7. Method
  8. FedAvg-Semi: A Strong Aggregation Baseline for Fedsemi
  9. SemiAnAgg: Semi-Supervised Anchor-Based Aggregation
  10. Experiments
  11. Experimental Setup
  12. Quantitative Comparisons with State-of-the-Art Methods
  13. Ablation studies
  14. Conclusion
  15. Limitations and Future Directions
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Upper: FedAvg McMahan2017CommunicationEfficientLO_fedavg fails miserably. Yet, disentangling the aggregation (FedAvg-Semi) achieves performance comparable to the state-of-the-art FedSemi method, CBAFed Li2023ClassBA_cbfed. By promoting diversity among unlabeled clients our SemiAnAgg achieves a new SOTA on four FedSemi benchmarks. Lower: Leave one unlabeled client out. The bar refers to the $\Delta$ Error (%), and the line refers to the Data Size. In the leave-one-out experiment setting, one unlabeled client is excluded during training. The results indicate that the decrease in accuracy (represented by the bar) does not correspond proportionally with the data size (represented by the line).
  • Figure 2: Illustration of the proposed Semi-supervised Anchor-Based Aggregation. The ignored pseudo label owing to low confidence is denoted as $-1$.
  • Figure 3: Ablation using FedAvg-Semi and SemiAnAgg with different random anchors. Upper: SVHN. Lower: ISIC-18.
  • Figure 4: SemiAnAgg convergence analysis on the SVHN dataset. Note that as training progresses, the unlabeled client has reliable pseudo labels in (a) ($\approx$ 99%) given digit classification is relatively a simple task. SemiAnAgg converges to the clients contributing almost equally 1/9 $\approx$ 0.1111. (c) Client Importance by leaving one unlabeled client out (Each bar corresponds to the performance drop in FedSemi where an unlabeled client is removed).
  • Figure 5: Performance improvement when the number of labeled clients is increased from one to two on ISIC.
  • ...and 7 more figures