FedDiv: Collaborative Noise Filtering for Federated Learning with Noisy Labels

TL;DR

FedDiv tackles Federated Learning with Noisy Labels (F-LNL) by introducing a one-stage framework that jointly learns a global Federated Noise Filter (FNF) and a Predictive Consistency based Sampler (PCS). The FNF aggregates per-client Gaussian Mixture Models fitted to per-sample losses to model the global noise distribution without sharing raw data, enabling robust identification of clean vs. noisy samples. PCS enforces agreement between local and global predictions and applies counterfactual debiasing to reselect high-quality labeled data for local training, reducing noise memorization and stabilizing optimization. Training employs MixUp with a class-prior regularization to further enhance robustness, yielding state-of-the-art results on CIFAR-10, CIFAR-100, and Clothing1M under both IID and non-IID partitions. Overall, FedDiv demonstrates that collaborative noise filtering, coupled with predictive-consistency data selection, can significantly improve FL performance in the presence of noisy labels while preserving data privacy.

Abstract

Federated learning with noisy labels (F-LNL) aims at seeking an optimal server model via collaborative distributed learning by aggregating multiple client models trained with local noisy or clean samples. On the basis of a federated learning framework, recent advances primarily adopt label noise filtering to separate clean samples from noisy ones on each client, thereby mitigating the negative impact of label noise. However, these prior methods do not learn noise filters by exploiting knowledge across all clients, leading to sub-optimal and inferior noise filtering performance and thus damaging training stability. In this paper, we present FedDiv to tackle the challenges of F-LNL. Specifically, we propose a global noise filter called Federated Noise Filter for effectively identifying samples with noisy labels on every client, thereby raising stability during local training sessions. Without sacrificing data privacy, this is achieved by modeling the global distribution of label noise across all clients. Then, in an effort to make the global model achieve higher performance, we introduce a Predictive Consistency based Sampler to identify more credible local data for local model training, thus preventing noise memorization and further boosting the training stability. Extensive experiments on CIFAR-10, CIFAR-100, and Clothing1M demonstrate that \texttt{FedDiv} achieves superior performance over state-of-the-art F-LNL methods under different label noise settings for both IID and non-IID data partitions. Source code is publicly available at https://github.com/lijichang/FLNL-FedDiv.

Figures (8)

• Figure 1: (a) Local noise filtering may have limited capabilities as each client develops its own local noise filter using its own private data only. Especially on clean clients, such filters would incorrectly identify a subset of clean samples to be noisy. (b) Collaborative noise filtering proposed by us significantly improves the performance of label noise filtering on each client as a federated noise filter is learned by distilling knowledge from all clients. PDF: Probability density function.
• Figure 2: An overview of the training procedure proposed by FedDiv. In this work, the parameters of a local neural model and a local noise filter are simultaneously learned on each client during the local training sessions, while both types of parameters are aggregated on the server.
• Figure 3: The accuracy of noisy label identification across different clients, with the experiment conducted on CIFAR-100 with the non-IID data partition under the noise setting $(\rho, \tau)=(0.4, 0.0)$. (Best viewed zoomed in.)
• Figure 4: The evolution of quantized training stability v.s. test classification performance across epochs for various F-LNL algorithms. We quantitatively assess training stability in F-LNL by computing the average proximal regularization metric FedProxxu2022fedcorr between the weights of local and global neural network models in the current training round. The experiments are conducted on CIFAR-10 with $(p, \alpha_{Dir})=(0.3, 10.0)$ and $(\rho, \tau)=(6.0, 0.5)$.
• Figure 5: The accuracy of noisy label identification vs. different clients. The lightblue dotted line represents the actual noise level of each client, while the (dotted) lines in deep bright colors indicate the noise filtering performance with respect to different noise filters. The experiment is conducted on CIFAR-100 with the IID data partition under the noise setting $(\rho, \tau)=(0.4, 0.5)$. TOP: Evaluation in the $50$-th communication round of the usual training stage; BOTTOM: Evaluation in the $500$-th communication round of the usual training stage.