Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning
Arnaud Descours, Léonard Deroose, Jan Ramon
TL;DR
This work tackles the communication bottleneck in federated learning by introducing ProjFL, which projects client gradients onto a subspace spanned by historical descent directions to enable highly compressed updates, and ProjFL+EF, which adds Error Feedback to handle biased compressors. The authors establish convergence guarantees across strongly convex, convex, and non-convex objectives, and demonstrate substantial practical gains, achieving up to an 8× reduction in communication while preserving accuracy on MNIST and CIFAR-10 with LeNet-5 and ResNet-20. The methods leverage a shared client-server subspace and minimal additional information per iteration, offering a principled, scalable approach to communication-efficient FL. These results have immediate impact for deploying FL in resource-constrained environments and large-scale models where communication is a critical bottleneck.
Abstract
Federated Learning (FL) enables decentralized model training across multiple clients while optionally preserving data privacy. However, communication efficiency remains a critical bottleneck, particularly for large-scale models. In this work, we introduce two complementary algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, tailored for biased compressors through an Error Feedback mechanism. Both methods rely on projecting local gradients onto a shared client-server subspace spanned by historical descent directions, enabling efficient information exchange with minimal communication overhead. We establish convergence guarantees for both algorithms under strongly convex, convex, and non-convex settings. Empirical evaluations on standard FL classification benchmarks with deep neural networks show that ProjFL and ProjFL+EF achieve accuracy comparable to existing baselines while substantially reducing communication costs.