Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Robust Clustered Federated Learning Approach for Non-IID Data with Quantity Skew

Michael Ben Ali, Imen Megdiche, André Peninou, Olivier Teste

TL;DR

The paper tackles the sensitivity of Clustered Federated Learning (CFL) to Quantity Skew (QS) in Non-IID FL. It first systematically evaluates state-of-the-art CFL methods under QS, revealing robustness gaps, then introduces CORNFLQS, a novel CFL algorithm that alternates weight-based and loss-based clustering to seek a common clustering agreement. Across 270 non-IID QS configurations on six image datasets, CORNFLQS achieves the highest average ranking in accuracy and clustering quality (ARI) and shows strong robustness to QS perturbations, outperforming existing CFL methods. This work highlights the importance of integrating model weights and losses for reliable client clustering in FL and provides a practical approach for real-world QS scenarios. Future work includes scaling to larger models and exploring privacy-preserving and efficient clustering variants.

Abstract

Federated Learning (FL) is a decentralized paradigm that enables a client-server architecture to collaboratively train a global Artificial Intelligence model without sharing raw data, thereby preserving privacy. A key challenge in FL is Non-IID data. Quantity Skew (QS) is a particular problem of Non-IID, where clients hold highly heterogeneous data volumes. Clustered Federated Learning (CFL) is an emergent variant of FL that presents a promising solution to Non-IID problem. It improves models' performance by grouping clients with similar data distributions into clusters. CFL methods generally fall into two operating strategies. In the first strategy, clients select the cluster that minimizes the local training loss. In the second strategy, the server groups clients based on local model similarities. However, most CFL methods lack systematic evaluation under QS but present significant challenges because of it. In this paper, we present two main contributions. The first one is an evaluation of state-of-the-art CFL algorithms under various Non-IID settings, applying multiple QS scenarios to assess their robustness. Our second contribution is a novel iterative CFL algorithm, named CORNFLQS, which proposes an optimal coordination between both operating strategies of CFL. Our approach is robust against the different variations of QS settings. We conducted intensive experiments on six image classification datasets, resulting in 270 Non-IID configurations. The results show that CORNFLQS achieves the highest average ranking in both accuracy and clustering quality, as well as strong robustness to QS perturbations. Overall, our approach outperforms actual CFL algorithms.

A Robust Clustered Federated Learning Approach for Non-IID Data with Quantity Skew

TL;DR

The paper tackles the sensitivity of Clustered Federated Learning (CFL) to Quantity Skew (QS) in Non-IID FL. It first systematically evaluates state-of-the-art CFL methods under QS, revealing robustness gaps, then introduces CORNFLQS, a novel CFL algorithm that alternates weight-based and loss-based clustering to seek a common clustering agreement. Across 270 non-IID QS configurations on six image datasets, CORNFLQS achieves the highest average ranking in accuracy and clustering quality (ARI) and shows strong robustness to QS perturbations, outperforming existing CFL methods. This work highlights the importance of integrating model weights and losses for reliable client clustering in FL and provides a practical approach for real-world QS scenarios. Future work includes scaling to larger models and exploring privacy-preserving and efficient clustering variants.

Abstract

Federated Learning (FL) is a decentralized paradigm that enables a client-server architecture to collaboratively train a global Artificial Intelligence model without sharing raw data, thereby preserving privacy. A key challenge in FL is Non-IID data. Quantity Skew (QS) is a particular problem of Non-IID, where clients hold highly heterogeneous data volumes. Clustered Federated Learning (CFL) is an emergent variant of FL that presents a promising solution to Non-IID problem. It improves models' performance by grouping clients with similar data distributions into clusters. CFL methods generally fall into two operating strategies. In the first strategy, clients select the cluster that minimizes the local training loss. In the second strategy, the server groups clients based on local model similarities. However, most CFL methods lack systematic evaluation under QS but present significant challenges because of it. In this paper, we present two main contributions. The first one is an evaluation of state-of-the-art CFL algorithms under various Non-IID settings, applying multiple QS scenarios to assess their robustness. Our second contribution is a novel iterative CFL algorithm, named CORNFLQS, which proposes an optimal coordination between both operating strategies of CFL. Our approach is robust against the different variations of QS settings. We conducted intensive experiments on six image classification datasets, resulting in 270 Non-IID configurations. The results show that CORNFLQS achieves the highest average ranking in both accuracy and clustering quality, as well as strong robustness to QS perturbations. Overall, our approach outperforms actual CFL algorithms.

Paper Structure

This paper contains 13 sections, 6 figures, 7 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. CFL for Non-IID Data
  3. Taxonomy of Data Heterogeneity in Federated Learning
  4. Non-IID Data with QS in CFL
  5. Comparative Evaluation of CFL Methods in QS Scenarios
  6. Federated Learning Setup
  7. Impact of Quantity Skew on CFL algorithms
  8. A Robust Solution for Non-IID Data with Quantity Skew
  9. Experimental results and discussion on CORNFLQS
  10. Effectiveness
  11. Robustness
  12. Sensitivity to clustering hyperparameter
  13. Conclusion

Figures (6)

  • Figure 1: A FL setup where all clients with the same images rotations have QS between them (Quantity Skew Type 1)
  • Figure 2: Another FL setup where 100 clients with different images rotations have QS between them (Quantity Skew Type 2)
  • Figure 4: Illustration of Non-IID categories for two clients $i$ and $j$ with samples from the MNIST dataset.
  • Figure 5: $\Delta$ ARI Heatmaps of CFL algorithms between non-QS and QS setups.
  • Figure 6: Winrate matrix based on accuracy across all experiments of CFL algorithms
  • ...and 1 more figures