Orthogonal Calibration for Asynchronous Federated Learning
Jiayun Zhang, Shuheng Li, Haiyu Huang, Xiaofan Yu, Rajesh K. Gupta, Jingbo Shang
TL;DR
OrthoFL tackles the inefficiencies of asynchronous federated learning under data heterogeneity by decoupling global and client optimization through orthogonal calibration. It projects the global weight shift accrued during client delays onto the subspace orthogonal to the receiving client's update, then merges the calibrated global shift into the client’s local training via a moving-average global update. The approach yields robust improvements in accuracy and convergence speed across diverse datasets and delay patterns, outperforming both synchronous and existing asynchronous baselines. This method enables more scalable and reliable federated learning in real-world, heterogeneous environments by reducing interference between global progress and local adaptation.
Abstract
Asynchronous federated learning mitigates the inefficiency of conventional synchronous aggregation by integrating updates as they arrive and adjusting their influence based on staleness. Due to asynchrony and data heterogeneity, learning objectives at the global and local levels are inherently inconsistent -- global optimization trajectories may conflict with ongoing local updates. Existing asynchronous methods simply distribute the latest global weights to clients, which can overwrite local progress and cause model drift. In this paper, we propose OrthoFL, an orthogonal calibration framework that decouples global and local learning progress and adjusts global shifts to minimize interference before merging them into local models. In OrthoFL, clients and the server maintain separate model weights. Upon receiving an update, the server aggregates it into the global weights via a moving average. For client weights, the server computes the global weight shift accumulated during the client's delay and removes the components aligned with the direction of the received update. The resulting parameters lie in a subspace orthogonal to the client update and preserve the maximal information from the global progress. The calibrated global shift is then merged into the client weights for further training. Extensive experiments show that OrthoFL improves accuracy by 9.6% and achieves a 12$\times$ speedup compared to synchronous methods. Moreover, it consistently outperforms state-of-the-art asynchronous baselines under various delay patterns and heterogeneity scenarios.