Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients

Tian Wen, Zhiqin Yang, Yonggang Zhang, Xuefeng Jiang, Hao Peng, Yuwei Wang, Bo Han

Abstract

Federated learning (FL) suffers from performance degradation due to the inevitable presence of noisy annotations in distributed scenarios. Existing approaches have advanced in distinguishing noisy samples from the dataset for label correction by leveraging loss values. However, noisy samples recognition relying on scalar loss lacks reliability for FL under heterogeneous scenarios. In this paper, we rethink this paradigm from a representation perspective and propose \method~(\textbf{Fed}erated under \textbf{R}epresentation \textbf{G}emometry), which follows \textbf{the principle of ``representation geometry priority''} to recognize noisy labels. Firstly, \method~creates label-agnostic spherical representations by using self-supervision. It then iteratively fits a spherical von Mises-Fisher (vMF) mixture model to this geometry using previously identified clean samples to capture semantic clusters. This geometric evidence is integrated with a semantic-label soft mapping mechanism to derive a distribution divergence between the label-free and annotated label-conditioned feature space, which robustly identifies noisy samples and updates the vMF mixture model with the newly separated clean dataset. Lastly, we employ an additional personalized noise absorption matrix on noisy labels to achieve robust optimization. Extensive experimental results demonstrate that \method~significantly outperforms state-of-the-art methods for FL with data heterogeneity under diverse noisy clients scenarios.

FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients

Abstract

Federated learning (FL) suffers from performance degradation due to the inevitable presence of noisy annotations in distributed scenarios. Existing approaches have advanced in distinguishing noisy samples from the dataset for label correction by leveraging loss values. However, noisy samples recognition relying on scalar loss lacks reliability for FL under heterogeneous scenarios. In this paper, we rethink this paradigm from a representation perspective and propose \method~(\textbf{Fed}erated under \textbf{R}epresentation \textbf{G}emometry), which follows \textbf{the principle of ``representation geometry priority''} to recognize noisy labels. Firstly, \method~creates label-agnostic spherical representations by using self-supervision. It then iteratively fits a spherical von Mises-Fisher (vMF) mixture model to this geometry using previously identified clean samples to capture semantic clusters. This geometric evidence is integrated with a semantic-label soft mapping mechanism to derive a distribution divergence between the label-free and annotated label-conditioned feature space, which robustly identifies noisy samples and updates the vMF mixture model with the newly separated clean dataset. Lastly, we employ an additional personalized noise absorption matrix on noisy labels to achieve robust optimization. Extensive experimental results demonstrate that \method~significantly outperforms state-of-the-art methods for FL with data heterogeneity under diverse noisy clients scenarios.
Paper Structure (25 sections, 10 equations, 4 figures, 2 tables)

This paper contains 25 sections, 10 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning
  4. Noisy Label Learning
  5. Noisy Labels in Federated Learning
  6. Methodology
  7. Preliminary
  8. Federated Learning.
  9. Federated Label Noise.
  10. Motivation
  11. Label Decoupled Spherical Representation
  12. Representation Geometry Priority Principle
  13. Geometric Representation via vMF Distribution
  14. Label-free Geometry Representation.
  15. Annotated Label-conditioned Geometry Representation.
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Comparison of noise identification performance between the existing method and FedRG under severe data heterogeneity. The existing methods mainly identify based on the distribution of scalar loss values. In heterogeneous scenarios, the loss value alone becomes unreliable, while FedRG provides more robust spatial geometry to filter out noise samples. Higher FN (False Negatives) but lower TP (True Positives) indicates that noise identification based on loss value wrongly classifies many correct labels as noise labels. The red circle denotes that the noisy samples we want to identify based on the geometric evidence.
  • Figure 2: The overview of FedRG which is consisted by two stage. The label decoupled spherical representation stage gets the hypersphere of samples. Then, we follow the representation geometry priority principle to split the clean and noisy samples based on the distribution divergence in the vMF distribution. The pseudocode overview of FedRG is provided in Appendix Sec.2
  • Figure 3: Comparison of the performance among different schemes under four heterogeneous noise scenarios. Each radar chart corresponds to a specific type of label noise: (a) symmetric with localized noise, (b) symmetric with globalized noise, (c) pairflip with localized noise, and (d) pairflip with globalized noise.
  • Figure 4: Comparison of model performance under different numbers of clients and semantic clusters.