Robust Model Aggregation for Heterogeneous Federated Learning: Analysis and Optimizations
Yumeng Shao, Jun Li, Long Shi, Kang Wei, Ming Ding, Qianmu Li, Zengxiang Li, Wen Chen, Shi Jin
TL;DR
The paper tackles latency and inconsistency in federated learning on heterogeneous devices by introducing Time-driven SFL (T-SFL), which aggregates at fixed time intervals with varying local iterations across clients. It derives a loss upper bound under T-SFL and optimizes aggregation weights to minimize this bound, then introduces the Discriminative Model Select (DMS) algorithm to filter underperforming clients and bias aggregation toward higher-quality updates. Empirical evaluations on MNIST, CIFAR-10, Fashion-MNIST, and SVHN show that T-SFL with DMS can cut training latency by about 50% and achieve up to 7% accuracy gains over AFL baselines, with competitive performance versus FedAvg/FedProx. The approach provides a robust, scalable FL framework for edge scenarios with dynamic participation and resource heterogeneity, supported by convergence guarantees and comprehensive experiments.
Abstract
Conventional synchronous federated learning (SFL) frameworks suffer from performance degradation in heterogeneous systems due to imbalanced local data size and diverse computing power on the client side. To address this problem, asynchronous FL (AFL) and semi-asynchronous FL have been proposed to recover the performance loss by allowing asynchronous aggregation. However, asynchronous aggregation incurs a new problem of inconsistency between local updates and global updates. Motivated by the issues of conventional SFL and AFL, we first propose a time-driven SFL (T-SFL) framework for heterogeneous systems. The core idea of T-SFL is that the server aggregates the models from different clients, each with varying numbers of iterations, at regular time intervals. To evaluate the learning performance of T-SFL, we provide an upper bound on the global loss function. Further, we optimize the aggregation weights to minimize the developed upper bound. Then, we develop a discriminative model selection (DMS) algorithm that removes local models from clients whose number of iterations falls below a predetermined threshold. In particular, this algorithm ensures that each client's aggregation weight accurately reflects its true contribution to the global model update, thereby improving the efficiency and robustness of the system. To validate the effectiveness of T-SFL with the DMS algorithm, we conduct extensive experiments using several popular datasets including MNIST, Cifar-10, Fashion-MNIST, and SVHN. The experimental results demonstrate that T-SFL with the DMS algorithm can reduce the latency of conventional SFL by 50\%, while achieving an average 3\% improvement in learning accuracy over state-of-the-art AFL algorithms.