Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible

TL;DR

This paper tackles the stale-update problem in asynchronous federated learning by introducing FedADT, a method that uses server-side knowledge distillation to align outdated client models with the latest global model and an adaptive weight to modulate guidance during training. FedADT employs a small distillation dataset and a distillation loss L_KD to transfer knowledge from the current global model to clients, while alpha_tilde(t) gradually increases the influence of distillation across training rounds. Experimental results on MNIST, FMNIST, CIFAR-10, and CIFAR-100 show that FedADT achieves faster convergence and higher final accuracy than existing asynchronous methods, with robustness to data heterogeneity and higher concurrency. This approach offers a practical path to robust, fast asynchronous federated learning with limited additional computation and communication overhead.

Abstract

As an emerging paradigm of federated learning, asynchronous federated learning offers significant speed advantages over traditional synchronous federated learning. Unlike synchronous federated learning, which requires waiting for all clients to complete updates before aggregation, asynchronous federated learning aggregates the models that have arrived in realtime, greatly improving training speed. However, this mechanism also introduces the issue of client model version inconsistency. When the differences between models of different versions during aggregation become too large, it may lead to conflicts, thereby reducing the models accuracy. To address this issue, this paper proposes an asynchronous federated learning version correction algorithm based on knowledge distillation, named FedADT. FedADT applies knowledge distillation before aggregating gradients, using the latest global model to correct outdated information, thus effectively reducing the negative impact of outdated gradients on the training process. Additionally, FedADT introduces an adaptive weighting function that adjusts the knowledge distillation weight according to different stages of training, helps mitigate the misleading effects caused by the poorer performance of the global model in the early stages of training. This method significantly improves the overall performance of asynchronous federated learning without adding excessive computational overhead. We conducted experimental comparisons with several classical algorithms, and the results demonstrate that FedADT achieves significant improvements over other asynchronous methods and outperforms all methods in terms of convergence speed.

Figures (6)

• Figure 1: The comparison of the experimental data distribution and algorithm performance. In the left figure, different colors of points represent different categories, and the position of each point reflects the proportion of each category's data in the corresponding client. In the figure on the right, 'Efficiency' refers to the ratio of the time taken by the optimal algorithm to reach a target accuracy of 35% to the time taken by each algorithm to achieve the same level of accuracy.
• Figure 2: FedADT Framework. Faster client models are almost always trained based on the latest global model, whereas slower client models can only be trained on outdated global models. Traditional methods typically aggregate these outdated models directly with the latest model, which can lead to conflicts between different model versions. In contrast, our approach leverages knowledge distillation to rapidly align outdated client models with the latest version, significantly reducing conflicts between different model versions.
• Figure 3: MNIST
• Figure 4: FMNIST
• Figure 5: CIFAR10