Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

(FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning

Seungjoo Lee, Thanh-Long V. Le, Jaemin Shin, Sung-Ju Lee

TL;DR

This work proposes $(FL)^2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization, and shows that regularizing the original pseudo-labeling loss is suboptimal, and hence it is carefully select unlabeled samples for regularization.

Abstract

Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data. However, most FL approaches assume that clients possess labeled data, which is often not the case in practice. Federated Semi-Supervised Learning (FSSL) addresses this label deficiency problem, targeting situations where only the server has a small amount of labeled data while clients do not. However, a significant performance gap exists between Centralized Semi-Supervised Learning (SSL) and FSSL. This gap arises from confirmation bias, which is more pronounced in FSSL due to multiple local training epochs and the separation of labeled and unlabeled data. We propose $(FL)^2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization. We show that regularizing the original pseudo-labeling loss is suboptimal, and hence we carefully select unlabeled samples for regularization. We further introduce client-specific adaptive thresholding and learning status-aware aggregation to adjust the training process based on the learning progress of each client. Our experiments on three benchmark datasets demonstrate that our approach significantly improves performance and bridges the gap with SSL, particularly in scenarios with scarce labeled data.

(FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning

TL;DR

This work proposes , a robust training method for unlabeled clients using sharpness-aware consistency regularization, and shows that regularizing the original pseudo-labeling loss is suboptimal, and hence it is carefully select unlabeled samples for regularization.

Abstract

Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data. However, most FL approaches assume that clients possess labeled data, which is often not the case in practice. Federated Semi-Supervised Learning (FSSL) addresses this label deficiency problem, targeting situations where only the server has a small amount of labeled data while clients do not. However, a significant performance gap exists between Centralized Semi-Supervised Learning (SSL) and FSSL. This gap arises from confirmation bias, which is more pronounced in FSSL due to multiple local training epochs and the separation of labeled and unlabeled data. We propose , a robust training method for unlabeled clients using sharpness-aware consistency regularization. We show that regularizing the original pseudo-labeling loss is suboptimal, and hence we carefully select unlabeled samples for regularization. We further introduce client-specific adaptive thresholding and learning status-aware aggregation to adjust the training process based on the learning progress of each client. Our experiments on three benchmark datasets demonstrate that our approach significantly improves performance and bridges the gap with SSL, particularly in scenarios with scarce labeled data.

Paper Structure

This paper contains 17 sections, 15 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Semi-supervised learning (SSL)
  4. Federated semi-supervised learning (FSSL)
  5. Preliminaries
  6. Federated learning
  7. Federated semi-supervised learning
  8. Sharpness-aware minimization
  9. Method
  10. Client-specific adaptive thresholding
  11. Sharpness-aware consistency regularization
  12. Learning status-aware aggregation
  13. Algorithm
  14. More experiment results
  15. Details of learning setup
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Comparison of SSL and FSSL algorithms on CIFAR-10 with varying numbers of labeled samples, where FreeMatch wang2022freematch represents SSL, and SemiFL diao2022semifl, FedCon long2021fedcon, and FedMatch jeong2020federated represent FSSL.
  • Figure 2: Overview of (FL)$^2$: (1) client-specific adaptive thresholding adjusts the pseudo-labeling threshold according to each client's learning status, (2) sharpness-aware consistency regularization ensures consistency between the original model and the adversarially perturbed model with carefully selected high-confident pseudo labels, and (3) learning status-aware aggregation aggregates client models considering each client's learning progress.