Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heterogeneous Federated Learning via Personalized Generative Networks

Zahra Taghiyarrenani, Abdallah Alabdallah, Slawomir Nowaczyk, Sepideh Pashami

TL;DR

The paper addresses statistical heterogeneity in Federated Learning by reframing the problem through domain adaptation theory and introducing FedGenP, which trains per-client generators on the server to produce latent-space synthetic data that reduce cross-client conflict. The approach provides theoretical guarantees via generalization bounds that relate client performance to distributional distances and empirical risks, and it demonstrates empirical gains on CelebA, MNIST, and EMNIST, especially when client data is scarce or highly heterogeneous. Key contributions include the server-side learning of personalized latent-space generators and a two-stage FedGenP training protocol that preserves data privacy while improving the global model's generalization. The work has practical implications for robust, privacy-preserving heterogeneous FL and suggests avenues for extending to clustered FL and more scalable server-side generation strategies.

Abstract

Federated Learning (FL) allows several clients to construct a common global machine-learning model without having to share their data. FL, however, faces the challenge of statistical heterogeneity between the client's data, which degrades performance and slows down the convergence toward the global model. In this paper, we provide theoretical proof that minimizing heterogeneity between clients facilitates the convergence of a global model for every single client. This becomes particularly important under empirical concept shifts among clients, rather than merely considering imbalanced classes, which have been studied until now. Therefore, we propose a method for knowledge transfer between clients where the server trains client-specific generators. Each generator generates samples for the corresponding client to remove the conflict with other clients' models. Experiments conducted on synthetic and real data, along with a theoretical study, support the effectiveness of our method in constructing a well-generalizable global model by reducing the conflict between local models.

Heterogeneous Federated Learning via Personalized Generative Networks

TL;DR

The paper addresses statistical heterogeneity in Federated Learning by reframing the problem through domain adaptation theory and introducing FedGenP, which trains per-client generators on the server to produce latent-space synthetic data that reduce cross-client conflict. The approach provides theoretical guarantees via generalization bounds that relate client performance to distributional distances and empirical risks, and it demonstrates empirical gains on CelebA, MNIST, and EMNIST, especially when client data is scarce or highly heterogeneous. Key contributions include the server-side learning of personalized latent-space generators and a two-stage FedGenP training protocol that preserves data privacy while improving the global model's generalization. The work has practical implications for robust, privacy-preserving heterogeneous FL and suggests avenues for extending to clustered FL and more scalable server-side generation strategies.

Abstract

Federated Learning (FL) allows several clients to construct a common global machine-learning model without having to share their data. FL, however, faces the challenge of statistical heterogeneity between the client's data, which degrades performance and slows down the convergence toward the global model. In this paper, we provide theoretical proof that minimizing heterogeneity between clients facilitates the convergence of a global model for every single client. This becomes particularly important under empirical concept shifts among clients, rather than merely considering imbalanced classes, which have been studied until now. Therefore, we propose a method for knowledge transfer between clients where the server trains client-specific generators. Each generator generates samples for the corresponding client to remove the conflict with other clients' models. Experiments conducted on synthetic and real data, along with a theoretical study, support the effectiveness of our method in constructing a well-generalizable global model by reducing the conflict between local models.
Paper Structure (16 sections, 15 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 16 sections, 15 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Notations and Preliminaries
  4. Federated Learning.
  5. Theoretical Motivation
  6. Symmetric difference hypothesis space $\mathcal{H}\Delta\mathcal{H}$ blitzer2007learning:
  7. $\mathcal{H}$-distance between two distributions blitzer2007learning:
  8. Lemma 1. Generalization Bounds for DA ben2006analysis:
  9. Theorem 1. Generalization Bounds for clients in FL:
  10. Our proposed method
  11. Server-Side Stage:
  12. Client-Side Stage:
  13. Experiments
  14. Discussion and Future Work
  15. Conclusion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Overview of the FedGenP method: Server constructs a global model together with client-specific generators and sends them to the clients. The clients (re)train their local models using both the original and generated data, and then send the trained local model to the server.
  • Figure 2: Illustration of the intuition behind generator's training process and data generation.
  • Figure 3: The progress of convergence of the global model on the CelebA dataset with r=1.
  • Figure 4: First row: Three different clients with totally different data distributions. Second row (from left): FedAvg after 40 communication steps, FedAvg after 100 communication steps, and FedGenP after 40 communication steps.