Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration

Huan Wang, Haoran Li, Huaming Chen, Jun Yan, Lijuan Wang, Jiahua Shi, Shiping Chen, Jun Shen

TL;DR

FedSKC tackles non-IID federated learning by extracting and transferring class-wise structural knowledge across clients. It combines Local Contrastive Learning, Global Discrepancy Aggregation, and Global Period Review to address local, global, and sampling drifts, respectively, providing a privacy-preserving supervisory signal from inter-client distributions. Theoretical analysis establishes non-convex convergence guarantees and rate results, while extensive experiments on non-IID, long-tailed, and few-shot settings demonstrate consistent improvements over strong baselines. The approach yields faster convergence, higher accuracy, and robust performance in practical FL scenarios, offering a scalable way to leverage class-wise structure in distributed learning.

Abstract

With the advancement of edge computing, federated learning (FL) displays a bright promise as a privacy-preserving collaborative learning paradigm. However, one major challenge for FL is the data heterogeneity issue, which refers to the biased labeling preferences among multiple clients, negatively impacting convergence and model performance. Most previous FL methods attempt to tackle the data heterogeneity issue locally or globally, neglecting underlying class-wise structure information contained in each client. In this paper, we first study how data heterogeneity affects the divergence of the model and decompose it into local, global, and sampling drift sub-problems. To explore the potential of using intra-client class-wise structural knowledge in handling these drifts, we thus propose Federated Learning with Structural Knowledge Collaboration (FedSKC). The key idea of FedSKC is to extract and transfer domain preferences from inter-client data distributions, offering diverse class-relevant knowledge and a fair convergent signal. FedSKC comprises three components: i) local contrastive learning, to prevent weight divergence resulting from local training; ii) global discrepancy aggregation, which addresses the parameter deviation between the server and clients; iii) global period review, correcting for the sampling drift introduced by the server randomly selecting devices. We have theoretically analyzed FedSKC under non-convex objectives and empirically validated its superiority through extensive experimental results.

FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration

TL;DR

FedSKC tackles non-IID federated learning by extracting and transferring class-wise structural knowledge across clients. It combines Local Contrastive Learning, Global Discrepancy Aggregation, and Global Period Review to address local, global, and sampling drifts, respectively, providing a privacy-preserving supervisory signal from inter-client distributions. Theoretical analysis establishes non-convex convergence guarantees and rate results, while extensive experiments on non-IID, long-tailed, and few-shot settings demonstrate consistent improvements over strong baselines. The approach yields faster convergence, higher accuracy, and robust performance in practical FL scenarios, offering a scalable way to leverage class-wise structure in distributed learning.

Abstract

With the advancement of edge computing, federated learning (FL) displays a bright promise as a privacy-preserving collaborative learning paradigm. However, one major challenge for FL is the data heterogeneity issue, which refers to the biased labeling preferences among multiple clients, negatively impacting convergence and model performance. Most previous FL methods attempt to tackle the data heterogeneity issue locally or globally, neglecting underlying class-wise structure information contained in each client. In this paper, we first study how data heterogeneity affects the divergence of the model and decompose it into local, global, and sampling drift sub-problems. To explore the potential of using intra-client class-wise structural knowledge in handling these drifts, we thus propose Federated Learning with Structural Knowledge Collaboration (FedSKC). The key idea of FedSKC is to extract and transfer domain preferences from inter-client data distributions, offering diverse class-relevant knowledge and a fair convergent signal. FedSKC comprises three components: i) local contrastive learning, to prevent weight divergence resulting from local training; ii) global discrepancy aggregation, which addresses the parameter deviation between the server and clients; iii) global period review, correcting for the sampling drift introduced by the server randomly selecting devices. We have theoretically analyzed FedSKC under non-convex objectives and empirically validated its superiority through extensive experimental results.

Paper Structure

This paper contains 19 sections, 3 theorems, 30 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Local Model Adjustment
  4. Global Model Adjustment
  5. Contrastive Learning
  6. Methodology
  7. Preliminaries
  8. Proposed FedSKC
  9. Theoretical Analysis
  10. Discussion
  11. Experiments
  12. Experimental Setup
  13. Evaluation Results
  14. Validation Analysis
  15. Conclusion
  16. ...and 4 more sections

Key Result

Theorem 1

With the assumptions, FedSKC loss function $\mathcal{L}$ of an arbitrary client will be bounded as follows:

Figures (4)

  • Figure 1: (a) An example of various drift sub-problems (local, global, and sampling drift) in FL training on round $r$; (b) Loss landscape visualization of two local models (left) and a global model (middle), and displaying the sampling drift of different rounds (right), where the coloured blocks represent the amount of data with different classes.
  • Figure 2: The overview of Federated Learning with Structural Knowledge Collaboration (FedSKC).
  • Figure 3: Effects of four key arguments ($\tau$ in Eq. \ref{['eq3']}, $\mathcal{M}$ in Eq. \ref{['eq1']}, $\alpha$, and $\rho$) in our proposed FedSKC.
  • Figure 4: Analysis for the local and global structural knowledge (Sec. \ref{['sec31']}) in our proposed FedSKC.

Theorems & Definitions (3)

  • Theorem 1: Deviation bound of the objective function
  • Theorem 2: Non-convex convergence of the FedSKC
  • Theorem 3: Non-convex convergence rate of the FedSKC