FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Data

TL;DR

FedAC tackles non-IID data in federated learning by proposing an adaptive clustered FL framework that decouples models into embedding and decision submodules to fuse global and intra-cluster knowledge, and employs a low-rank cosine similarity metric for efficient online clustering. It introduces a bi-level objective with local loss, intra-cluster regularization, and global embedding regularization, solved via alternating minimization, and adds an EM-like re-clustering step plus a cluster number tuning module to adapt K during training. Empirical evaluation on CIFAR-10 and CIFAR-100 under non-IID settings shows FedAC consistently surpassing SOTA CFL methods in accuracy, validating the effectiveness of global knowledge infusion, LrCos-based clustering, and dynamic K. The approach enhances robustness and scalability in heterogeneous environments, making CFL more practical for real-world deployments.

Abstract

Clustered federated learning (CFL) is proposed to mitigate the performance deterioration stemming from data heterogeneity in federated learning (FL) by grouping similar clients for cluster-wise model training. However, current CFL methods struggle due to inadequate integration of global and intra-cluster knowledge and the absence of an efficient online model similarity metric, while treating the cluster count as a fixed hyperparameter limits flexibility and robustness. In this paper, we propose an adaptive CFL framework, named FedAC, which (1) efficiently integrates global knowledge into intra-cluster learning by decoupling neural networks and utilizing distinct aggregation methods for each submodule, significantly enhancing performance; (2) includes a costeffective online model similarity metric based on dimensionality reduction; (3) incorporates a cluster number fine-tuning module for improved adaptability and scalability in complex, heterogeneous environments. Extensive experiments show that FedAC achieves superior empirical performance, increasing the test accuracy by around 1.82% and 12.67% on CIFAR-10 and CIFAR-100 datasets, respectively, under different non-IID settings compared to SOTA methods.

Figures (4)

• Figure 1: The framwork of FedAC in the heterogeneous setting. The server clusters and aggregates clients to get a global embedding and multiple cluster center models. Clients update local models by minimizing the classification error loss ($L_{\text{sup}}$), intra-clustering regularization ($L_{\text{intra}}$), and global regularization ($L_{\text{global}}$).
• Figure 2: Clients employed FedAvg with CIFAR-10 CIFAR10 for $20$ epochs, succeeded by local fine-tuning, with model similarities subsequently evaluated.
• Figure 3: Ablation experiments were conducted to assess FedAC's performance across different parameter configurations and model similarity measurements.
• Figure 4: Client models, trained with FeSEM and FedAC, are visualized by projecting them onto a plane in (a) and (b). Clients with similar data distributions are represented by similar colors. In (c), the mean and variance of clusters' $G_c$ are shown for various predefined $K$ values (with $K=10$ as the optimal value), while subplot (d) depicts the CNT module fine-tuning the cluster number by adjusting $G_c$ (marked in black).