BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
Heng Zhu, Qing Ling
TL;DR
The paper tackles the dual challenge of communication efficiency and Byzantine robustness in federated learning by analyzing how gradient compression interacts with robust aggregation. It shows that vanilla Byzantine-robust compressed SGD deteriorates under compression noise and stochastic noise, motivating a noise-reduction strategy. The authors introduce BROADCAST, which combines gradient difference compression with SAGA variance reduction to mitigate both sources of noise, and provide theoretical guarantees of linear convergence to a neighborhood with error comparable to uncompressed methods. Empirical results on logistic regression and neural networks demonstrate BROADCAST's effectiveness against multiple Byzantine attack types, outperforming existing compressed-robust baselines. This work advances practical, reliable, and communication-efficient distributed learning in adversarial environments.
Abstract
Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale federated learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed federated learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.