Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiConfederated Learning: Inclusive Non-IID Data handling with Decentralized Federated Learning

Michael Duchesne, Kaiwen Zhang, Chamseddine Talhi

TL;DR

This work introduces MultiConfederated Learning (MCFL), a decentralized Federated Learning framework designed to address non-IID data by maintaining multiple parallel models, enabling forks, and allowing transfer learning between groups. MCFL eliminates the central aggregator, uses a DAG-based organization, and employs weight-divergence-aware update selection to improve convergence and personalization across sovereign data owners. Empirical results on IID and non-IID MNIST/FashionMNIST tasks show MCFL can outperform FedAvg in several scenarios, offering faster convergence, better robustness to data skew, and the ability to converge to a single model when beneficial. The approach provides a flexible, privacy-preserving path toward robust decentralized learning with practical implications for cross-silo, privacy-conscious deployments.

Abstract

Federated Learning (FL) has emerged as a prominent privacy-preserving technique for enabling use cases like confidential clinical machine learning. FL operates by aggregating models trained by remote devices which owns the data. Thus, FL enables the training of powerful global models using crowd-sourced data from a large number of learners, without compromising their privacy. However, the aggregating server is a single point of failure when generating the global model. Moreover, the performance of the model suffers when the data is not independent and identically distributed (non-IID data) on all remote devices. This leads to vastly different models being aggregated, which can reduce the performance by as much as 50% in certain scenarios. In this paper, we seek to address the aforementioned issues while retaining the benefits of FL. We propose MultiConfederated Learning: a decentralized FL framework which is designed to handle non-IID data. Unlike traditional FL, MultiConfederated Learning will maintain multiple models in parallel (instead of a single global model) to help with convergence when the data is non-IID. With the help of transfer learning, learners can converge to fewer models. In order to increase adaptability, learners are allowed to choose which updates to aggregate from their peers.

MultiConfederated Learning: Inclusive Non-IID Data handling with Decentralized Federated Learning

TL;DR

This work introduces MultiConfederated Learning (MCFL), a decentralized Federated Learning framework designed to address non-IID data by maintaining multiple parallel models, enabling forks, and allowing transfer learning between groups. MCFL eliminates the central aggregator, uses a DAG-based organization, and employs weight-divergence-aware update selection to improve convergence and personalization across sovereign data owners. Empirical results on IID and non-IID MNIST/FashionMNIST tasks show MCFL can outperform FedAvg in several scenarios, offering faster convergence, better robustness to data skew, and the ability to converge to a single model when beneficial. The approach provides a flexible, privacy-preserving path toward robust decentralized learning with practical implications for cross-silo, privacy-conscious deployments.

Abstract

Federated Learning (FL) has emerged as a prominent privacy-preserving technique for enabling use cases like confidential clinical machine learning. FL operates by aggregating models trained by remote devices which owns the data. Thus, FL enables the training of powerful global models using crowd-sourced data from a large number of learners, without compromising their privacy. However, the aggregating server is a single point of failure when generating the global model. Moreover, the performance of the model suffers when the data is not independent and identically distributed (non-IID data) on all remote devices. This leads to vastly different models being aggregated, which can reduce the performance by as much as 50% in certain scenarios. In this paper, we seek to address the aforementioned issues while retaining the benefits of FL. We propose MultiConfederated Learning: a decentralized FL framework which is designed to handle non-IID data. Unlike traditional FL, MultiConfederated Learning will maintain multiple models in parallel (instead of a single global model) to help with convergence when the data is non-IID. With the help of transfer learning, learners can converge to fewer models. In order to increase adaptability, learners are allowed to choose which updates to aggregate from their peers.
Paper Structure (22 sections, 4 equations, 6 figures, 2 tables, 2 algorithms)

This paper contains 22 sections, 4 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Non-IID data
  4. Federated Learning
  5. Background
  6. Federated Learning
  7. Decentralized Machine Learning
  8. Non-IID Data
  9. Multitask Learning
  10. Proposed solutions
  11. System Model
  12. Network Initialisation and Learning Process
  13. Updates Aggregation
  14. Forks Management
  15. Communication Costs
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of the three types of machine learning network deployments.
  • Figure 2: Evolution of models during a single MultiConfederated Learning round.
  • Figure 3: An overview of a MultiConfederated Learning network comprising 5 nodes, which are divided into 3 distinct groups.
  • Figure 4: Accuracy of learners using FedAvg vs. MultiConfederated Learning.
  • Figure 5: Average accuracy of multiple update selection tolerances with data distributed with no clear groups
  • ...and 1 more figures