FNBench: Benchmarking Robust Federated Learning against Noisy Labels

TL;DR

FNBench provides the first unified benchmark for evaluating robustness of federated learning under noisy labels, covering three noise patterns across five image datasets and one text dataset with eighteen competing methods. The study reveals that no single method dominates under all Non-IID and noise conditions, highlights memorization and dimensional collapse as core failure modes, and introduces a representation-aware regularization term, L_SVD, which improves robustness across several baselines. By combining this regularization with existing methods, the benchmark demonstrates consistent gains and insights into managing noisy labels in FL. The work also discusses practical trade-offs, limitations, and directions for real-world deployment, and releases open-source code to facilitate further research and fair comparisons.

Abstract

Robustness to label noise within data is a significant challenge in federated learning (FL). From the data-centric perspective, the data quality of distributed datasets can not be guaranteed since annotations of different clients contain complicated label noise of varying degrees, which causes the performance degradation. There have been some early attempts to tackle noisy labels in FL. However, there exists a lack of benchmark studies on comprehensively evaluating their practical performance under unified settings. To this end, we propose the first benchmark study FNBench to provide an experimental investigation which considers three diverse label noise patterns covering synthetic label noise, imperfect human-annotation errors and systematic errors. Our evaluation incorporates eighteen state-of-the-art methods over five image recognition datasets and one text classification dataset. Meanwhile, we provide observations to understand why noisy labels impair FL, and additionally exploit a representation-aware regularization method to enhance the robustness of existing methods against noisy labels based on our observations. Finally, we discuss the limitations of this work and propose three-fold future directions. To facilitate related communities, our source code is open-sourced at https://github.com/Sprinter1999/FNBench.

Figures (7)

• Figure 1: Clients in FL are of various noise rates. Clients of higher noise rates tend to yield worse models and further affect the convergence and final performance of the aggregated global model. Many Robust FL methods aim to mitigate the negative effects caused by these high-noise clients via more cautious model aggregation at the server side.
• Figure 2: The training paradigm of FL FedLSRfedavg. Local datasets are often Non-IID, and they can contain data with noisy labels which hampers the global model's convergence and final performance.
• Figure 3: Noise transition matrices of different synthetic noise types (using 5 classes as example). Green grids denote the percentage that is correctly labeled to the ground truth class while the red denotes the wrongly labeled percentage to other classes.
• Figure 4: Noise transition matrices for CIFAR-10/100-N. Human-annotation noise transition patterns are more complex than synthetic noise transition patterns in Figure \ref{['fig:case']}. For more information, please kindly refer to cifarn.
• Figure 5: Test accuracy curves on both IID and Non-IID data. (a) illustrates the ideal case where clients have the same noise rate on IID data, which aligns the settings of FedLSR FedLSR. Note that the noise rates across clients in (b) and (c) are inconsistent.