Tackling Resource-Constrained and Data-Heterogeneity in Federated Learning with Double-Weight Sparse Pack

TL;DR

FedCSPACK tackles the dual challenges of data heterogeneity and resource-constrained clients in federated learning by packaging model parameters into Top-K cosine-based packs and applying a mask with dual weights (directional via cosine similarity and distribution via KL divergence) during aggregation. This design reduces communication load while preserving or enhancing global model generalization across non-IID clients. Empirical results on four datasets show strong accuracy gains, substantial communication savings, and robustness to limited client participation, outperforming ten SOTA baselines. The approach offers a practical, scalable path for personalized FL on devices with limited bandwidth and computation, achieving faster convergence and stable performance across diverse data distributions.

Abstract

Federated learning has drawn widespread interest from researchers, yet the data heterogeneity across edge clients remains a key challenge, often degrading model performance. Existing methods enhance model compatibility with data heterogeneity by splitting models and knowledge distillation. However, they neglect the insufficient communication bandwidth and computing power on the client, failing to strike an effective balance between addressing data heterogeneity and accommodating limited client resources. To tackle this limitation, we propose a personalized federated learning method based on cosine sparsification parameter packing and dual-weighted aggregation (FedCSPACK), which effectively leverages the limited client resources and reduces the impact of data heterogeneity on model performance. In FedCSPACK, the client packages model parameters and selects the most contributing parameter packages for sharing based on cosine similarity, effectively reducing bandwidth requirements. The client then generates a mask matrix anchored to the shared parameter package to improve the alignment and aggregation efficiency of sparse updates on the server. Furthermore, directional and distribution distance weights are embedded in the mask to implement a weighted-guided aggregation mechanism, enhancing the robustness and generalization performance of the global model. Extensive experiments across four datasets using ten state-of-the-art methods demonstrate that FedCSPACK effectively improves communication and computational efficiency while maintaining high model accuracy.

Figures (10)

• Figure 1: FedCSPACK Overview. The client shows the detailed process of the generation and selection of the local parameters package and mask. The server side displays the details of the dual-weighted aggregation.
• Figure 2: Data heterogeneity of 5 clients is simulated using dirichlet sampling ($Dir(0.6)$) on CIFAR-10.
• Figure 3: The generalization of the global model on the clients in CIFAR-10 and Dir(0.3).
• Figure 4: The influence of the limited resources client participation ratio (CPR) on CIFAR-100, where CPR represents the ratio of the number of clients per round to the total number of clients. The horizontal line indicates the accuracy of the FedCSPACK.
• Figure 5: The impact of the $PACK$ size on model performance and computation time.