Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedPAE: Peer-Adaptive Ensemble Learning for Asynchronous and Model-Heterogeneous Federated Learning

Brianna Mueller, W. Nick Street, Stephen Baek, Qihang Lin, Jingyi Yang, Yankun Huang

TL;DR

This work introduces Federated Peer-Adaptive Ensemble Learning (FedPAE), a fully decentralized pFL algorithm that supports model heterogeneity and asynchronous learning and utilizes a peer-to-peer model sharing mechanism and ensemble selection to achieve a more refined balance between local and global information.

Abstract

Federated learning (FL) enables multiple clients with distributed data sources to collaboratively train a shared model without compromising data privacy. However, existing FL paradigms face challenges due to heterogeneity in client data distributions and system capabilities. Personalized federated learning (pFL) has been proposed to mitigate these problems, but often requires a shared model architecture and a central entity for parameter aggregation, resulting in scalability and communication issues. More recently, model-heterogeneous FL has gained attention due to its ability to support diverse client models, but existing methods are limited by their dependence on a centralized framework, synchronized training, and publicly available datasets. To address these limitations, we introduce Federated Peer-Adaptive Ensemble Learning (FedPAE), a fully decentralized pFL algorithm that supports model heterogeneity and asynchronous learning. Our approach utilizes a peer-to-peer model sharing mechanism and ensemble selection to achieve a more refined balance between local and global information. Experimental results show that FedPAE outperforms existing state-of-the-art pFL algorithms, effectively managing diverse client capabilities and demonstrating robustness against statistical heterogeneity.

FedPAE: Peer-Adaptive Ensemble Learning for Asynchronous and Model-Heterogeneous Federated Learning

TL;DR

This work introduces Federated Peer-Adaptive Ensemble Learning (FedPAE), a fully decentralized pFL algorithm that supports model heterogeneity and asynchronous learning and utilizes a peer-to-peer model sharing mechanism and ensemble selection to achieve a more refined balance between local and global information.

Abstract

Federated learning (FL) enables multiple clients with distributed data sources to collaboratively train a shared model without compromising data privacy. However, existing FL paradigms face challenges due to heterogeneity in client data distributions and system capabilities. Personalized federated learning (pFL) has been proposed to mitigate these problems, but often requires a shared model architecture and a central entity for parameter aggregation, resulting in scalability and communication issues. More recently, model-heterogeneous FL has gained attention due to its ability to support diverse client models, but existing methods are limited by their dependence on a centralized framework, synchronized training, and publicly available datasets. To address these limitations, we introduce Federated Peer-Adaptive Ensemble Learning (FedPAE), a fully decentralized pFL algorithm that supports model heterogeneity and asynchronous learning. Our approach utilizes a peer-to-peer model sharing mechanism and ensemble selection to achieve a more refined balance between local and global information. Experimental results show that FedPAE outperforms existing state-of-the-art pFL algorithms, effectively managing diverse client capabilities and demonstrating robustness against statistical heterogeneity.

Paper Structure

This paper contains 14 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Federated Learning with non-IID data
  4. Personalized Federated Learning
  5. Model-heterogeneous federated learning
  6. Ensemble Selection
  7. Methods
  8. Peer-Adaptive Ensemble Learning
  9. Model Selection and Ensemble Evaluation
  10. Experimental Design
  11. Results
  12. Discussion
  13. Limitations
  14. Conclusion

Figures (5)

  • Figure 1: A centralized and synchronous FL network
  • Figure 2: Overview of Peer-Adaptive Ensemble Learning
  • Figure 3: Example Pareto front for a single client generated from the final generation of NSGA-II. Each point represents a potential ensemble, with the axes showing the trade-off between ensemble strength and diversity.
  • Figure 4: Client data distributions on CIFAR-10 at three levels of statistical heterogeneity. The x-axis indicates client IDs, the y-axis indicates class labels, and the size of each point represents the number of data samples. As the Dirichlet distribution parameter decreases from 0.5 (left) to 0.1 (right), statistical heterogeneity increases. Higher heterogeneity results in clients having more samples concentrated in fewer class labels.
  • Figure 5: Performance on CIFAR-10 and CIFAR-100 across different levels of heterogeneity