Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedHQ: Hybrid Runtime Quantization for Federated Learning

Zihao Zheng, Ziyao Wang, Xiuping Cui, Maoliang Li, Jiayu Chen, Yun, Liang, Ang Li, Xiang Chen

TL;DR

Federated Learning protects data privacy but suffers efficiency bottlenecks. The authors introduce FedHQ, a hybrid runtime quantization framework that automatically dispatches PTQ and QAT across clients using hardware- and data-driven analyses. FedHQ combines a coarse global initialization with a fine-grained ML-based adjustment to balance speed and accuracy in diverse FL settings. Empirical results show up to 2.47x training acceleration and up to 11.15% accuracy gains with negligible extra overhead, highlighting FedHQ's practical impact for scalable FL.

Abstract

Federated Learning (FL) is a decentralized model training approach that preserves data privacy but struggles with low efficiency. Quantization, a powerful training optimization technique, has been widely explored for integration into FL. However, many studies fail to consider the distinct performance attribution between particular quantization strategies, such as post-training quantization (PTQ) or quantization-aware training (QAT). As a result, existing FL quantization methods rely solely on either PTQ or QAT, optimizing for speed or accuracy while compromising the other. To efficiently accelerate FL and maintain distributed convergence accuracy across various FL settings, this paper proposes a hybrid quantitation approach combining PTQ and QAT for FL systems. We conduct case studies to validate the effectiveness of using hybrid quantization in FL. To solve the difficulty of modeling speed and accuracy caused by device and data heterogeneity, we propose a hardware-related analysis and data-distribution-related analysis to help identify the trade-off boundaries for strategy selection. Based on these, we proposed a novel framework named FedHQ to automatically adopt optimal hybrid strategy allocation for FL systems. Specifically, FedHQ develops a coarse-grained global initialization and fine-grained ML-based adjustment to ensure efficiency and robustness. Experiments show that FedHQ achieves up to 2.47x times training acceleration and up to 11.15% accuracy improvement and negligible extra overhead.

FedHQ: Hybrid Runtime Quantization for Federated Learning

TL;DR

Federated Learning protects data privacy but suffers efficiency bottlenecks. The authors introduce FedHQ, a hybrid runtime quantization framework that automatically dispatches PTQ and QAT across clients using hardware- and data-driven analyses. FedHQ combines a coarse global initialization with a fine-grained ML-based adjustment to balance speed and accuracy in diverse FL settings. Empirical results show up to 2.47x training acceleration and up to 11.15% accuracy gains with negligible extra overhead, highlighting FedHQ's practical impact for scalable FL.

Abstract

Federated Learning (FL) is a decentralized model training approach that preserves data privacy but struggles with low efficiency. Quantization, a powerful training optimization technique, has been widely explored for integration into FL. However, many studies fail to consider the distinct performance attribution between particular quantization strategies, such as post-training quantization (PTQ) or quantization-aware training (QAT). As a result, existing FL quantization methods rely solely on either PTQ or QAT, optimizing for speed or accuracy while compromising the other. To efficiently accelerate FL and maintain distributed convergence accuracy across various FL settings, this paper proposes a hybrid quantitation approach combining PTQ and QAT for FL systems. We conduct case studies to validate the effectiveness of using hybrid quantization in FL. To solve the difficulty of modeling speed and accuracy caused by device and data heterogeneity, we propose a hardware-related analysis and data-distribution-related analysis to help identify the trade-off boundaries for strategy selection. Based on these, we proposed a novel framework named FedHQ to automatically adopt optimal hybrid strategy allocation for FL systems. Specifically, FedHQ develops a coarse-grained global initialization and fine-grained ML-based adjustment to ensure efficiency and robustness. Experiments show that FedHQ achieves up to 2.47x times training acceleration and up to 11.15% accuracy improvement and negligible extra overhead.
Paper Structure (20 sections, 7 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 20 sections, 7 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Quantization Strategy in FL
  4. Challenges of Hybrid Quantization in FL
  5. Analysis of Quantization Strategy in FL
  6. Speed Analysis of Quantization Strategy in FL
  7. Accuracy Analysis of Quantization Strategy in FL
  8. Speed-Accuracy Trade-off Boundary
  9. Hybrid Runtime Quantization Framework
  10. Overview
  11. Coarse-grained Global Initialization
  12. Fine-grained ML-based Adjustment
  13. Convergence Analysis
  14. Experiment
  15. Setup
  16. ...and 5 more sections

Figures (5)

  • Figure 1: An Overview of Different Quantization Strategy in FL: (a) QAT Strategy in FL; (b) PTQ Strategy in FL.
  • Figure 2: FL quantization Analysis: (a) FL Settings; (b) Speed Analysis; (c) Accuracy Analysis.
  • Figure 3: Speed-accuracy Trade-off Boundary.
  • Figure 4: An Overview of FedHQ.
  • Figure 5: Accuracy Results of FedHQ and Baselines.