Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Early-Learning Regularization When Federated Learning Meets Noisy Labels

Taehyeon Kim, Donggyu Kim, Se-Young Yun

TL;DR

Early-learning regularization is revisits, introducing an innovative strategy, Federated Label-mixture Regularization (FLR), that adeptly adapts to FL's complexities by generating new pseudo labels, blending local and global model predictions.

Abstract

In the evolving landscape of federated learning (FL), addressing label noise presents unique challenges due to the decentralized and diverse nature of data collection across clients. Traditional centralized learning approaches to mitigate label noise are constrained in FL by privacy concerns and the heterogeneity of client data. This paper revisits early-learning regularization, introducing an innovative strategy, Federated Label-mixture Regularization (FLR). FLR adeptly adapts to FL's complexities by generating new pseudo labels, blending local and global model predictions. This method not only enhances the accuracy of the global model in both i.i.d. and non-i.i.d. settings but also effectively counters the memorization of noisy labels. Demonstrating compatibility with existing label noise and FL techniques, FLR paves the way for improved generalization in FL environments fraught with label inaccuracies.

Revisiting Early-Learning Regularization When Federated Learning Meets Noisy Labels

TL;DR

Early-learning regularization is revisits, introducing an innovative strategy, Federated Label-mixture Regularization (FLR), that adeptly adapts to FL's complexities by generating new pseudo labels, blending local and global model predictions.

Abstract

In the evolving landscape of federated learning (FL), addressing label noise presents unique challenges due to the decentralized and diverse nature of data collection across clients. Traditional centralized learning approaches to mitigate label noise are constrained in FL by privacy concerns and the heterogeneity of client data. This paper revisits early-learning regularization, introducing an innovative strategy, Federated Label-mixture Regularization (FLR). FLR adeptly adapts to FL's complexities by generating new pseudo labels, blending local and global model predictions. This method not only enhances the accuracy of the global model in both i.i.d. and non-i.i.d. settings but also effectively counters the memorization of noisy labels. Demonstrating compatibility with existing label noise and FL techniques, FLR paves the way for improved generalization in FL environments fraught with label inaccuracies.
Paper Structure (49 sections, 7 equations, 13 figures, 7 tables, 1 algorithm)

This paper contains 49 sections, 7 equations, 13 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Data Heterogeneity
  4. Noise Level
  5. Noise Type
  6. Method
  7. In the View of Memorization in FL
  8. Local & Global Moving Average Label Mixture
  9. Tackling Confirmation Bias
  10. Dissecting Gradients
  11. Experimental Results
  12. Analysis according to Data Heterogeneity
  13. Qualitative Analysis
  14. Implementation
  15. Implications of $\alpha, \beta, \gamma$
  16. ...and 34 more sections

Figures (13)

  • Figure 1: Overview of our proposed regularization, Federated Label-mixture Regularization (FLR).
  • Figure 2: (a) Server side memorization with $\mathcal{L}_{CE}$, and (b) client side memorization with $\mathcal{L}_{CE}$ on CIFAR-10 of the i.i.d. setting, with symmetric noise of $(\rho,\tau)=(0.8,0.0)$. In (b), the fraction values are calculated by averaging the values contributed solely by participating noisy clients in each round.
  • Figure 3: (a) Server-side memorization with $\mathcal{L}_{FLR}$, and (b) client-side memorization with $\mathcal{L}_{FLR}$ under the same setting used in \ref{['fig:general_memorization']}.
  • Figure 4: t-SNE mapping view of our latent features. Colors represent the noisy label $y$ and the ground truth label $y^*$.
  • Figure 5: (a) Heatmaps with respect to $\alpha,\gamma$ at $(\rho, \tau) = (0.6, 0.5)$ on CIFAR-10, (b) learning curves according to the changes of $\beta$ at $(\rho, \tau) = (1.0, 0.5)$ on CIFAR-10.
  • ...and 8 more figures