Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedFQ: Federated Learning with Fine-Grained Quantization

Haowei Li, Weiying Xie, Hangyu Ye, Jitao Ma, Shuran Ma, Yunsong Li

TL;DR

FedFQ addresses the communication bottleneck in federated learning by introducing fine-grained, parameter-level adaptive quantization. A Constraint-Guided Simulated Annealing (CGSA) solver designs the per-parameter bit-widths under a fixed budget, yielding an unbiased quantizer with a variance bound $q_f$ that can be much smaller than conventional quantizers, thereby mitigating quantization-induced drift on Non-IID data. The authors establish convergence guarantees for both strongly convex and non-convex objectives and demonstrate, through extensive experiments on CIFAR-10 and Shakespeare, that FedFQ achieves $\$27\\times$ to $63\\times$ uplink compression with lossless accuracy and faster convergence in large-scale, realistic network settings. The work combines rigorous theory with practical optimization and empirical validation, highlighting FedFQ’s potential to enable scalable, communication-efficient FL in real-world deployments.

Abstract

Federated learning (FL) is a decentralized approach, enabling multiple participants to collaboratively train a model while ensuring the protection of data privacy. The transmission of updates from numerous edge clusters to the server creates a significant communication bottleneck in FL. Quantization is an effective compression technology, showcasing immense potential in addressing this bottleneck problem. The Non-IID nature of FL renders it sensitive to quantization. Existing quantized FL frameworks inadequately balance high compression ratios and superior convergence performance by roughly employing a uniform quantization bit-width on the client-side. In this work, we propose a communication-efficient FL algorithm with a fine-grained adaptive quantization strategy (FedFQ). FedFQ addresses the trade-off between achieving high communication compression ratios and maintaining superior convergence performance by introducing parameter-level quantization. Specifically, we have designed a Constraint-Guided Simulated Annealing algorithm to determine specific quantization schemes. We derive the convergence of FedFQ, demonstrating its superior convergence performance compared to existing quantized FL algorithms. We conducted extensive experiments on multiple benchmarks and demonstrated that, while maintaining lossless performance, FedFQ achieves a compression ratio of 27 times to 63 times compared to the baseline experiment.

FedFQ: Federated Learning with Fine-Grained Quantization

TL;DR

FedFQ addresses the communication bottleneck in federated learning by introducing fine-grained, parameter-level adaptive quantization. A Constraint-Guided Simulated Annealing (CGSA) solver designs the per-parameter bit-widths under a fixed budget, yielding an unbiased quantizer with a variance bound that can be much smaller than conventional quantizers, thereby mitigating quantization-induced drift on Non-IID data. The authors establish convergence guarantees for both strongly convex and non-convex objectives and demonstrate, through extensive experiments on CIFAR-10 and Shakespeare, that FedFQ achieves 27\\times63\\times$ uplink compression with lossless accuracy and faster convergence in large-scale, realistic network settings. The work combines rigorous theory with practical optimization and empirical validation, highlighting FedFQ’s potential to enable scalable, communication-efficient FL in real-world deployments.

Abstract

Federated learning (FL) is a decentralized approach, enabling multiple participants to collaboratively train a model while ensuring the protection of data privacy. The transmission of updates from numerous edge clusters to the server creates a significant communication bottleneck in FL. Quantization is an effective compression technology, showcasing immense potential in addressing this bottleneck problem. The Non-IID nature of FL renders it sensitive to quantization. Existing quantized FL frameworks inadequately balance high compression ratios and superior convergence performance by roughly employing a uniform quantization bit-width on the client-side. In this work, we propose a communication-efficient FL algorithm with a fine-grained adaptive quantization strategy (FedFQ). FedFQ addresses the trade-off between achieving high communication compression ratios and maintaining superior convergence performance by introducing parameter-level quantization. Specifically, we have designed a Constraint-Guided Simulated Annealing algorithm to determine specific quantization schemes. We derive the convergence of FedFQ, demonstrating its superior convergence performance compared to existing quantized FL algorithms. We conducted extensive experiments on multiple benchmarks and demonstrated that, while maintaining lossless performance, FedFQ achieves a compression ratio of 27 times to 63 times compared to the baseline experiment.
Paper Structure (14 sections, 5 theorems, 14 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 5 theorems, 14 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fine-grained And Adaptive Quantization Strategies
  4. FedAvg Formulation
  5. The Convergence of FedFQ
  6. Quantization Strategy Of FedFQ
  7. Experiments
  8. Experiment Setup
  9. Ablation Experiment
  10. Comparative Experiment
  11. The Experiments On CIFAR-10
  12. The Experiments On Shakespeare
  13. Evaluation in the Real-world Network Environment
  14. Conclusion

Key Result

Lemma 1

(Unbiasness and Bounded Variance of Random Uniform Quantization): The random quantizer $Q(\cdot)$ is unbiased and its variance grows with the squared of l2-norm of its argument, i.e.,

Figures (5)

  • Figure 1: The schematic diagram of FedFQ. $\boldsymbol{h}$ represents local model updates. FedFQ's fine-grained quantization is an adaptive quantization technique targeting individual parameters in the parameter space, as opposed to the existing methods of uniformly quantizing the entire parameter space.
  • Figure 2: The learning curves of FedFQ and single-precision quantization.
  • Figure 3: Performance comparison between FedFQ and comparison algorithms on SimpleCNN.
  • Figure 4: Performance comparison between FedFQ and comparison algorithms on VGG11.
  • Figure 5: Performance comparison between FedFQ and comparison algorithms on LSTM.

Theorems & Definitions (5)

  • Lemma 1
  • Theorem 2
  • Corollary 3
  • Theorem 4
  • Corollary 5