Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

TL;DR

The paper tackles the communication bottleneck and straggler problem in Federated Learning under bandwidth heterogeneity and non-IID data. It introduces Bandwidth-aware Compression Ratio Scheduling (BCRS) to dynamically adjust compression ratios and client-averaging coefficients based on bandwidth, and Overlap-aware Parameter Weighted Average (OPWA) to reweight parameter updates according to their distribution across clients. The approach formalizes the FL objective $F(w)=\sum_{k=1}^N p_k F_k(w)$ and standard FedAvg updates, while evaluating on CIFAR-10/100 and SVHN with Dirichlet non-IID partitions to demonstrate up to 13% accuracy gains and up to $2.02$–$3.37\times$ speedups over baselines like Top-K. The results indicate strong improvements in both convergence speed and final model accuracy, offering a practical, modular framework for cross-device, communication-efficient FL in heterogeneous environments.

Abstract

Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID (Independently and Identically Distributed) data. To address these issues, we introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data. First, our strategy dynamically adjusts compression ratios according to bandwidth, enabling clients to upload their models at a close pace, thus exploiting the otherwise wasted time to transmit more data. Second, we identify the non-overlapped pattern of retained parameters after compression, which results in diminished client update signals due to uniformly averaged weights. Based on this finding, we propose a parameter mask to adjust the client-averaging coefficients at the parameter level, thereby more closely approximating the original updates, and improving the training convergence under heterogeneous environments. Our evaluations reveal that our method significantly boosts model accuracy, with a maximum improvement of 13% over the uncompressed FedAvg. Moreover, it achieves a $3.37\times$ speedup in reaching the target accuracy compared to FedAvg with a Top-K compressor, demonstrating its effectiveness in accelerating convergence with compression. The integration of common compression techniques into our framework further establishes its potential as a versatile foundation for future cross-device, communication-efficient FL research, addressing critical challenges in FL and advancing the field of distributed machine learning.

Figures (15)

• Figure 1: Timelines of different methods with FedAvg. Comm. represents communication, C1, C2, C3 represent three different clients. $B_1 > B_2 > B_3$ for these clients.
• Figure 2: Adaptive communication ratios based on client bandwidth $B_1 > B_2 > B_3$. Such methods make clients 1 and 2 retain as much information as possible while guaranteeing the communication time will not be larger than the uniform compression.
• Figure 3: Illustration of Parameter Overlap: Smaller magnitude of less overlapped parameters compared to overlapped parameters after averaging. Param represents the model parameters, C1, C2, and C3 represent different clients.
• Figure 4: Distribution of degree of overlap of retained parameters after compression.
• Figure 5: NIID Distribution Across Clients for CIFAR-10.