FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure

TL;DR

FedGroup tackles statistical heterogeneity in Federated Learning by clustering clients according to the direction of their local optimization updates using a data-driven distance called Euclidean distance of Decomposed Cosine similarity ($\text{EDC}$). The framework introduces group-level training with intra-group FedAvg and optional inter-group aggregation (FedGrouProx adds a proximal term $\mu$), plus practical cold-start mechanisms for new and newcomer clients. Empirical results on MNIST, FEMNIST, Synthetic, and Sentiment140 demonstrate substantial accuracy gains over FedAvg, FedProx, and existing CFL methods, with FEMNIST gains reaching up to $+26.9\%$ over FeSEM and notable improvements on other datasets. The approach provides a scalable, HDLSS-friendly CFL paradigm with open-source implementation, enabling practical deployment in large-scale FL systems. All mathematical notation is presented with proper delimitation in $...$.

Abstract

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID and imbalanced (statistical heterogeneity) training data of FL is distributed in the federated network, which will increase the divergences between the local models and global model, further degrading performance. In this paper, we propose a novel clustered federated learning (CFL) framework FedGroup, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for high training performance; 2) construct a new data-driven distance measure to improve the efficiency of the client clustering procedure. 3) implement a newcomer device cold start mechanism based on the auxiliary global model for framework scalability and practicality. FedGroup can achieve improvements by dividing joint optimization into groups of sub-optimization and can be combined with FL optimizer FedProx. The convergence and complexity are analyzed to demonstrate the efficiency of our proposed framework. We also evaluate FedGroup and FedGrouProx (combined with FedProx) on several open datasets and made comparisons with related CFL frameworks. The results show that FedGroup can significantly improve absolute test accuracy by +14.1% on FEMNIST compared to FedAvg. +3.4% on Sentiment140 compared to FedProx, +6.9% on MNIST compared to FeSEM.

Figures (5)

• Figure 1: A FedAvg training procedure on three non-IID MNIST datasets and one IID MNIST dataset to illustrate the effects of statistical heterogeneity on model accuracy and discrepancy. From left to right, the number of classes of training data per client increase, which means the degree of data heterogeneity decreases. The discrepancy is defined in Equation \ref{['for:discrepancy']}.
• Figure 2: An overview of FedGroup.
• Figure 3: Evaluation results on MNIST ($m=3$) and FEMNIST($m=5$). Top: test accuracy; Middle: mean training loss; Bottom: discrepancy between clients and server (FedAvg and FedProx) or weighted discrepancy between clients and groups (FedGroup, FedGrouProx, IFCA, FeSEM).
• Figure 4: Classification accuracy evaluated on Synthetic(1,1) ($m=5$) and Sent140 ($m=5$).
• Figure 5: The mapping from the MADC to the EDC, the pre-training scale $\alpha=20$.