Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning

Zhiqin Yang, Yonggang Zhang, Chenxin Li, Yiu-ming Cheung, Bo Han, Yixuan Yuan

TL;DR

This paper tackles data heterogeneity in federated learning by proposing FedGPS, a two-pronged framework that blends distribution-level surrogate alignment with gradient-level path rectification. The static objective re-frames local goals to implicitly model the global distribution using privacy-preserving surrogate data, while the dynamic path rectification injects non-self gradient information to steer local updates toward global consistency. Theoretical results bound the surrogate-distribution gap via a Wasserstein distance, and extensive experiments across CIFAR-10, CIFAR-100, and SVHN demonstrate robust, state-of-the-art performance across diverse heterogeneity scenarios, with ablations confirming the complementary value of both components. FedGPS emphasizes robustness and practicality for real-world FL deployments, offering a pathway to distribution-agnostic learning under privacy constraints.

Abstract

Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving performance in certain heterogeneity scenarios remains an overlooked question: \textit{How robust are these methods to deploy under diverse heterogeneity scenarios?} To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Meanwhile, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspective. Motivated by this, we propose \textbf{FedGPS} (\textbf{Fed}erated \textbf{G}oal-\textbf{P}ath \textbf{S}ynergy), a novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client's learning objective to implicitly model the global data distribution using surrogate information, while dynamically adjusting local update directions with gradient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: https://github.com/CUHK-AIM-Group/FedGPS.

FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning

TL;DR

This paper tackles data heterogeneity in federated learning by proposing FedGPS, a two-pronged framework that blends distribution-level surrogate alignment with gradient-level path rectification. The static objective re-frames local goals to implicitly model the global distribution using privacy-preserving surrogate data, while the dynamic path rectification injects non-self gradient information to steer local updates toward global consistency. Theoretical results bound the surrogate-distribution gap via a Wasserstein distance, and extensive experiments across CIFAR-10, CIFAR-100, and SVHN demonstrate robust, state-of-the-art performance across diverse heterogeneity scenarios, with ablations confirming the complementary value of both components. FedGPS emphasizes robustness and practicality for real-world FL deployments, offering a pathway to distribution-agnostic learning under privacy constraints.

Abstract

Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving performance in certain heterogeneity scenarios remains an overlooked question: \textit{How robust are these methods to deploy under diverse heterogeneity scenarios?} To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Meanwhile, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspective. Motivated by this, we propose \textbf{FedGPS} (\textbf{Fed}erated \textbf{G}oal-\textbf{P}ath \textbf{S}ynergy), a novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client's learning objective to implicitly model the global data distribution using surrogate information, while dynamically adjusting local update directions with gradient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: https://github.com/CUHK-AIM-Group/FedGPS.
Paper Structure (35 sections, 4 theorems, 42 equations, 15 figures, 19 tables, 1 algorithm)

This paper contains 35 sections, 4 theorems, 42 equations, 15 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Methodology
  5. Motivation
  6. Static Goal-oriented Objective
  7. Dynamic Path-oriented Rectification
  8. Experiments
  9. Evaluated Details
  10. Implementation Details
  11. Main Results
  12. Ablation Study on Two Perspectives
  13. Conclusion and Further Discussion
  14. Proof Results
  15. Proof of Necessary Lemma \ref{['lemma:parameter_bound']}
  16. ...and 20 more sections

Key Result

Theorem 4.4

Given the global feature distribution $\mathcal{P}_g$, the local feature distribution $\mathcal{P}_k$, the surrogate distributions $\mathcal{P}^s$ (global) and $\mathcal{P}^{s}_k$ (local for the $k$-th client) over their corresponding data space. Suppose there exists $\kappa\geq 0$ such that $W_1(\m where $W_1$ is the Wasserstein-1 distance as defined in Definition def:w_distance. then the Wassers

Figures (15)

  • Figure 1: Nemenyi post-hoc test results on the performance under (a) SVHN and (b) CIFAR-10. Black horizontal lines indicate the critical distance (CD).
  • Figure 2: (a) and (b) are examples of data distribution scenarios generated using the Dirichlet partition method under the CIFAR-10 dataset across 10 clients. All scenarios use the same heterogeneity control factor of $\alpha=0.1$, but vary the random seed to produce different heterogeneous distributions. (c) The divergence between local and global surrogate distributions is computed as the FL training proceeds with different ratios of client sampling, also means the proportion of the data that participates in the global update at each federated training round. The divergence is computed every 5 rounds.
  • Figure 3: (a) Static goal-oriented objective. This objective is composed of two stages: local distribution aligns with local surrogate distribution (Alignment at local), and local surrogate distribution aligns with global surrogate distribution (Alignment with global). (b) Dynamic path-oriented rectification corrects the original update direction by the gradient of other clients for a new update path.
  • Figure 4: The visualization of the ablation study and convergence of FedGPS compared with other baselines. Due to the large volume of baselines, we select the top 5 baselines to plot.
  • Figure 5: The visualization of the CIFAR-10 dataset distribution across $K=10$ clients under five different heterogeneous scenarios.
  • ...and 10 more figures

Theorems & Definitions (14)

  • Definition 3.1: Wasserstein Distance
  • Definition 4.1: Surrogate Dataset
  • Definition 4.2: Local Surrogate Distribution
  • Definition 4.3: Global Surrogate Distribution
  • Theorem 4.4
  • Remark 1
  • Definition 4.5: Non-Self Gradient at Round $t$ of client $i$, $\delta^t_{\bm{\theta}_i}$
  • Remark 2
  • Lemma A.2
  • Lemma A.3: Bounded difference between global and local feature extractor $\bm{\psi}$, $\bm{\psi}_k$
  • ...and 4 more