Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaboratively Learning Federated Models from Noisy Decentralized Data

Haoyuan Li, Mathias Funk, Nezihe Merve Gürel, Aaqib Saeed

TL;DR

Federated learning suffers when client data contain input-space noise, limiting global generalization. The authors introduce Federated Noise Sifting (FedNS), a gradient-norm based method that identifies noisy clients in a single interaction by analyzing $g(\mathbf{x})=\left\| \nabla_{\theta} \mathcal{L}(\theta;\mathbf{x},\mathbf{y}) \right\|_{p}$ and then applies a noise-aware aggregation with weights $\alpha$ and $\beta$ to attenuate their influence, enabling plug-in integration with common FL strategies. The approach is formalized for noisy FL, demonstrated with a two-stage process (single-interaction discovery and plug-in aggregation), and validated across six datasets with four FL strategies under IID and non-IID conditions, including distortions, patch-based noise, and real-world label noise. Results show robust improvements in global model generalization, highlighting FedNS as a scalable, privacy-preserving tool for robust FL in decentralized data environments.

Abstract

Federated learning (FL) has emerged as a prominent method for collaboratively training machine learning models using local data from edge devices, all while keeping data decentralized. However, accounting for the quality of data contributed by local clients remains a critical challenge in FL, as local data are often susceptible to corruption by various forms of noise and perturbations, which compromise the aggregation process and lead to a subpar global model. In this work, we focus on addressing the problem of noisy data in the input space, an under-explored area compared to the label noise. We propose a comprehensive assessment of client input in the gradient space, inspired by the distinct disparity observed between the density of gradient norm distributions of models trained on noisy and clean input data. Based on this observation, we introduce a straightforward yet effective approach to identify clients with low-quality data at the initial stage of FL. Furthermore, we propose a noise-aware FL aggregation method, namely Federated Noise-Sifting (FedNS), which can be used as a plug-in approach in conjunction with widely used FL strategies. Our extensive evaluation on diverse benchmark datasets under different federated settings demonstrates the efficacy of FedNS. Our method effortlessly integrates with existing FL strategies, enhancing the global model's performance by up to 13.68% in IID and 15.85% in non-IID settings when learning from noisy decentralized data.

Collaboratively Learning Federated Models from Noisy Decentralized Data

TL;DR

Federated learning suffers when client data contain input-space noise, limiting global generalization. The authors introduce Federated Noise Sifting (FedNS), a gradient-norm based method that identifies noisy clients in a single interaction by analyzing and then applies a noise-aware aggregation with weights and to attenuate their influence, enabling plug-in integration with common FL strategies. The approach is formalized for noisy FL, demonstrated with a two-stage process (single-interaction discovery and plug-in aggregation), and validated across six datasets with four FL strategies under IID and non-IID conditions, including distortions, patch-based noise, and real-world label noise. Results show robust improvements in global model generalization, highlighting FedNS as a scalable, privacy-preserving tool for robust FL in decentralized data environments.

Abstract

Federated learning (FL) has emerged as a prominent method for collaboratively training machine learning models using local data from edge devices, all while keeping data decentralized. However, accounting for the quality of data contributed by local clients remains a critical challenge in FL, as local data are often susceptible to corruption by various forms of noise and perturbations, which compromise the aggregation process and lead to a subpar global model. In this work, we focus on addressing the problem of noisy data in the input space, an under-explored area compared to the label noise. We propose a comprehensive assessment of client input in the gradient space, inspired by the distinct disparity observed between the density of gradient norm distributions of models trained on noisy and clean input data. Based on this observation, we introduce a straightforward yet effective approach to identify clients with low-quality data at the initial stage of FL. Furthermore, we propose a noise-aware FL aggregation method, namely Federated Noise-Sifting (FedNS), which can be used as a plug-in approach in conjunction with widely used FL strategies. Our extensive evaluation on diverse benchmark datasets under different federated settings demonstrates the efficacy of FedNS. Our method effortlessly integrates with existing FL strategies, enhancing the global model's performance by up to 13.68% in IID and 15.85% in non-IID settings when learning from noisy decentralized data.
Paper Structure (10 sections, 6 equations, 8 figures, 9 tables)

This paper contains 10 sections, 6 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Problem Definition
  5. Single-Interaction Discovery of Noisy Clients
  6. Federated Noise Sifting
  7. Experiments
  8. Datasets, Models, and Configuration
  9. Results and Ablation Studies
  10. Conclusion

Figures (8)

  • Figure 1: Overview of FedNS. We propose a plug-in noise-sensitive gradient-norm weighted federated aggregation approach for training high-quality deep models from decentralized data.
  • Figure 2: Impact of contrast noise on an image with different levels of noise severity.
  • Figure 3: Illustration of patch-based data corruption.
  • Figure 4: Gradient norm density during federated training on Fashion-MNIST (15 clean, 5 noisy clients, $\emph{NL}$=100%) using mean aggregation.
  • Figure 5: Qualitative analysis of clean and noisy CIFAR-10 samples identified by FedNS. Nine images from each cluster are shown. In Figure \ref{['subfig:data_sample_noisy']}, noisy samples exhibit mixed corruptions. Red and green grids denote noisy and clean images, respectively.
  • ...and 3 more figures