Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

Xinyuan Ji, Zhaowei Zhu, Wei Xi, Olga Gadyatskaya, Zilong Song, Yong Cai, Yang Liu

TL;DR

FedFixer addresses heterogeneous label noise in Federated Learning by deploying a dual-model framework consisting of a global model $w$ and a personalized model $\theta$ that mutually filter clean samples. It introduces two regularizers—Confidence Regularizer and Distance Regularizer—to curb overfitting and drift during local updates, enabling robust denoising across diverse clients. Extensive experiments on MNIST, CIFAR-10, and Clothing1M demonstrate competitive performance with SOTA methods and clear advantages in highly heterogeneous noise settings, along with improved denoising stability. This approach advances robust FL by explicitly handling client-specific noise patterns and leveraging cross-model cooperation for reliable data filtering in decentralized environments.

Abstract

Federated Learning (FL) heavily depends on label quality for its performance. However, the label distribution among individual clients is always both noisy and heterogeneous. The high loss incurred by client-specific samples in heterogeneous label noise poses challenges for distinguishing between client-specific and noisy label samples, impacting the effectiveness of existing label noise learning approaches. To tackle this issue, we propose FedFixer, where the personalized model is introduced to cooperate with the global model to effectively select clean client-specific samples. In the dual models, updating the personalized model solely at a local level can lead to overfitting on noisy data due to limited samples, consequently affecting both the local and global models' performance. To mitigate overfitting, we address this concern from two perspectives. Firstly, we employ a confidence regularizer to alleviate the impact of unconfident predictions caused by label noise. Secondly, a distance regularizer is implemented to constrain the disparity between the personalized and global models. We validate the effectiveness of FedFixer through extensive experiments on benchmark datasets. The results demonstrate that FedFixer can perform well in filtering noisy label samples on different clients, especially in highly heterogeneous label noise scenarios.

FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

TL;DR

FedFixer addresses heterogeneous label noise in Federated Learning by deploying a dual-model framework consisting of a global model and a personalized model that mutually filter clean samples. It introduces two regularizers—Confidence Regularizer and Distance Regularizer—to curb overfitting and drift during local updates, enabling robust denoising across diverse clients. Extensive experiments on MNIST, CIFAR-10, and Clothing1M demonstrate competitive performance with SOTA methods and clear advantages in highly heterogeneous noise settings, along with improved denoising stability. This approach advances robust FL by explicitly handling client-specific noise patterns and leveraging cross-model cooperation for reliable data filtering in decentralized environments.

Abstract

Federated Learning (FL) heavily depends on label quality for its performance. However, the label distribution among individual clients is always both noisy and heterogeneous. The high loss incurred by client-specific samples in heterogeneous label noise poses challenges for distinguishing between client-specific and noisy label samples, impacting the effectiveness of existing label noise learning approaches. To tackle this issue, we propose FedFixer, where the personalized model is introduced to cooperate with the global model to effectively select clean client-specific samples. In the dual models, updating the personalized model solely at a local level can lead to overfitting on noisy data due to limited samples, consequently affecting both the local and global models' performance. To mitigate overfitting, we address this concern from two perspectives. Firstly, we employ a confidence regularizer to alleviate the impact of unconfident predictions caused by label noise. Secondly, a distance regularizer is implemented to constrain the disparity between the personalized and global models. We validate the effectiveness of FedFixer through extensive experiments on benchmark datasets. The results demonstrate that FedFixer can perform well in filtering noisy label samples on different clients, especially in highly heterogeneous label noise scenarios.
Paper Structure (20 sections, 1 theorem, 10 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 20 sections, 1 theorem, 10 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust FL
  4. Learning with Noisy Labels
  5. Problem Definition
  6. FedFixer: Dual Model Structure Approach
  7. Dual Model Structure
  8. Regularizers
  9. Confidence Regularizer
  10. Distance Regularizer
  11. Experiments
  12. Experimental Setup
  13. Baselines
  14. Implementation Details
  15. Comparison with State-of-the-Art Methods
  16. ...and 5 more sections

Key Result

Theorem 1

The sample selector defined in (eq:sample_sieve) ensures that clean examples ($x_n$, $\tilde{y}_n = y_n$) will not be identified as noisy label samples if the model $f(\cdot)$’s prediction on $x_n$ is better than a random guess.

Figures (4)

  • Figure 1: (a): The heterogeneous label noise distributions encompass diverse true class samples or varying label noise transitions. (b): The performance comparison between CL and FL on CIFAR-10 with the rate of label noise 0.2, 0.5, and 0.8.
  • Figure 2: Overall architecture. The training process of the proposed FedFixer has three stages: ① deployment of the global model, ② the local model updates, and ③ the global model aggregation. In the second stage, dual models are alternately updated (in the Dual Model Update module) based on the selected samples (in the Filter Noisy Data module) by each other.
  • Figure 3: Clients' F-score distribution of different methods on CIFAR-10 with different noise levels on IID and non-IID distribution.
  • Figure 4: Loss distribution of clean and noisy samples training without CR and with CR on CIFAR-10 with noise model $\rho = 0.5, \tau = 0.3$ and IID data partition (Round = 20).

Theorems & Definitions (2)

  • Definition : Client-Dependent Label Noise
  • Theorem 1