Asynchronous Federated Learning: A Scalable Approach for Decentralized Machine Learning
Ali Forootani, Raffaele Iervolino
TL;DR
This work tackles the synchronization bottlenecks of traditional federated learning by proposing Asynchronous Federated Learning (AFL) with bounded delays $\tau_c \le \tau_{\max}$. It develops a convergence framework based on martingale difference sequences and variance bounds, deriving a recursion under $\mu$-strong convexity that accounts for client delays and stale gradients. The authors establish variance bounds for sampling without replacement and quantify client drift via a drift term, demonstrating robust convergence despite asynchrony. Empirical validation on convex tasks—decentralized linear regression and linear SVM with non-IID Dirichlet data—shows AFL's scalability and robustness, achieving competitive final performance and faster practical convergence under realistic network heterogeneity.
Abstract
Federated Learning (FL) has emerged as a powerful paradigm for decentralized machine learning, enabling collaborative model training across diverse clients without sharing raw data. However, traditional FL approaches often face limitations in scalability and efficiency due to their reliance on synchronous client updates, which can result in significant delays and increased communication overhead, particularly in heterogeneous and dynamic environments. To address these challenges in this paper, we propose an Asynchronous Federated Learning (AFL) algorithm, which allows clients to update the global model independently and asynchronously. Our key contributions include a comprehensive convergence analysis of AFL in the presence of client delays and model staleness. By leveraging martingale difference sequence theory and variance bounds, we ensure robust convergence despite asynchronous updates. Assuming strongly convex local objective functions, we establish bounds on gradient variance under random client sampling and derive a recursion formula quantifying the impact of client delays on convergence. Furthermore, we demonstrate the practical applicability of the AFL algorithm by training decentralized linear regression and Support Vector Machine (SVM) based classifiers and compare its results with synchronous FL algorithm to effectively handling non-IID data distributed among clients. The proposed AFL algorithm addresses key limitations of traditional FL methods, such as inefficiency due to global synchronization and susceptibility to client drift. It enhances scalability, robustness, and efficiency in real-world settings with heterogeneous client populations and dynamic network conditions. Our results underscore the potential of AFL to drive advancements indistributed learning systems, particularly for large-scale, privacy-preserving applications in resource-constrained environments.