Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

Chao Feng, Hongjie Guan, Alberto Huertas Celdrán, Jan von der Assen, Gérôme Bovet, Burkhard Stiller

TL;DR

This work tackles non-IID data in Decentralized Federated Learning (DFL) by introducing FedEP, an entropy-pooling-based aggregation method. FedEP first fits local data distributions with Gaussian Mixture Models (GMMs) and shares only distribution parameters, then estimates a global distribution and computes aggregation weights via KL-divergence between local and global distributions. The approach yields faster convergence and improved accuracy on MNIST, Fashion-MNIST, and CIFAR10 under diverse non-IID settings, while preserving data privacy and maintaining low communication overhead. Its privacy-preserving, scalable design makes FedEP a practical solution for heterogeneous data in large-scale decentralized FL environments.

Abstract

Non-Independent and Identically Distributed (non-IID) data in Federated Learning (FL) causes client drift issues, leading to slower convergence and reduced model performance. While existing approaches mitigate this issue in Centralized FL (CFL) using a central server, Decentralized FL (DFL) remains underexplored. In DFL, the absence of a central entity results in nodes accessing a global view of the federation, further intensifying the challenges of non-IID data. Drawing on the entropy pooling algorithm employed in financial contexts to synthesize diverse investment opinions, this work proposes the Federated Entropy Pooling (FedEP) algorithm to mitigate the non-IID challenge in DFL. FedEP leverages Gaussian Mixture Models (GMM) to fit local data distributions, sharing statistical parameters among neighboring nodes to estimate the global distribution. Aggregation weights are determined using the entropy pooling approach between local and global distributions. By sharing only synthetic distribution information, FedEP preserves data privacy while minimizing communication overhead. Experimental results demonstrate that FedEP achieves faster convergence and outperforms state-of-the-art methods in various non-IID settings.

FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

TL;DR

This work tackles non-IID data in Decentralized Federated Learning (DFL) by introducing FedEP, an entropy-pooling-based aggregation method. FedEP first fits local data distributions with Gaussian Mixture Models (GMMs) and shares only distribution parameters, then estimates a global distribution and computes aggregation weights via KL-divergence between local and global distributions. The approach yields faster convergence and improved accuracy on MNIST, Fashion-MNIST, and CIFAR10 under diverse non-IID settings, while preserving data privacy and maintaining low communication overhead. Its privacy-preserving, scalable design makes FedEP a practical solution for heterogeneous data in large-scale decentralized FL environments.

Abstract

Non-Independent and Identically Distributed (non-IID) data in Federated Learning (FL) causes client drift issues, leading to slower convergence and reduced model performance. While existing approaches mitigate this issue in Centralized FL (CFL) using a central server, Decentralized FL (DFL) remains underexplored. In DFL, the absence of a central entity results in nodes accessing a global view of the federation, further intensifying the challenges of non-IID data. Drawing on the entropy pooling algorithm employed in financial contexts to synthesize diverse investment opinions, this work proposes the Federated Entropy Pooling (FedEP) algorithm to mitigate the non-IID challenge in DFL. FedEP leverages Gaussian Mixture Models (GMM) to fit local data distributions, sharing statistical parameters among neighboring nodes to estimate the global distribution. Aggregation weights are determined using the entropy pooling approach between local and global distributions. By sharing only synthetic distribution information, FedEP preserves data privacy while minimizing communication overhead. Experimental results demonstrate that FedEP achieves faster convergence and outperforms state-of-the-art methods in various non-IID settings.

Paper Structure

This paper contains 16 sections, 16 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Improving Global Model
  4. Adapting Local Model
  5. FedEP Algorithm
  6. Problem Statement
  7. Algorithm Description
  8. Phase 1: Pre-train Distribution Fitting.
  9. Phase 2: Global Distribution Estimation.
  10. Phase 3: Entropy Pooling and Weighted Aggregation.
  11. Deployment and Experiments
  12. Experiments Configuration
  13. Pure Non-IID Scenario
  14. Mixed Non-IID Scenario
  15. Discussion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Overview of the FedEP Process, the BLUE phases are newly introduced by FedEP.
  • Figure 2: In the pure non-IID scenario, the F1-Score for each round for MNIST, Fashion-MNIST, CIFAR10
  • Figure 3: In the mixed non-IID scenario, the F1-Score for each round for MNIST, Fashion-MNIST, CIFAR10
  • Figure 4: In the 20-nodes pure non-IID scenario, the F1-Score for each round for MNIST, Fashion-MNIST, CIFAR10