Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$γ$-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning

Rongwei Lu, Yutong Jiang, Jinrui Zhang, Chunyang Li, Yifei Zhu, Bin Chen, Zhi Wang

TL;DR

The paper tackles the challenge of gradient compression in Federated Learning under non-IID data and decaying stepsize, where hard-threshold sparsification can hinder convergence. It introduces $\gamma$-FedHT, a stepsize-aware hard-threshold compressor with Error-Feedback that maintains $O(d)$ compression cost and achieves convergence rates matching FedAVG for both convex and non-convex objectives, with a compression-error term scaling as $\lambda_0^2$. The threshold is adaptively set via $\lambda_t^2 = \lambda_0^2 F(\gamma_t)$ using the function $F(\gamma_t) = (\gamma_t^{\alpha} + \gamma_t^{-\alpha})^{-1}$ and $\alpha=1$, ensuring the compression adapts to learning progress. Empirical results across four non-IID benchmarks show substantial gains over Top-$k$ under equal traffic, with improvements up to $7.42\%$ on CNN-CIFAR-$10$, demonstrating practical benefits for scalable, communication-efficient FL.

Abstract

Gradient compression can effectively alleviate communication bottlenecks in Federated Learning (FL). Contemporary state-of-the-art sparse compressors, such as Top-$k$, exhibit high computational complexity, up to $\mathcal{O}(d\log_2{k})$, where $d$ is the number of model parameters. The hard-threshold compressor, which simply transmits elements with absolute values higher than a fixed threshold, is thus proposed to reduce the complexity to $\mathcal{O}(d)$. However, the hard-threshold compression causes accuracy degradation in FL, where the datasets are non-IID and the stepsize $γ$ is decreasing for model convergence. The decaying stepsize reduces the updates and causes the compression ratio of the hard-threshold compression to drop rapidly to an aggressive ratio. At or below this ratio, the model accuracy has been observed to degrade severely. To address this, we propose $γ$-FedHT, a stepsize-aware low-cost compressor with Error-Feedback to guarantee convergence. Given that the traditional theoretical framework of FL does not consider Error-Feedback, we introduce the fundamental conversation of Error-Feedback. We prove that $γ$-FedHT has the convergence rate of $\mathcal{O}(\frac{1}{T})$ ($T$ representing total training iterations) under $μ$-strongly convex cases and $\mathcal{O}(\frac{1}{\sqrt{T}})$ under non-convex cases, \textit{same as FedAVG}. Extensive experiments demonstrate that $γ$-FedHT improves accuracy by up to $7.42\%$ over Top-$k$ under equal communication traffic on various non-IID image datasets.

$γ$-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning

TL;DR

The paper tackles the challenge of gradient compression in Federated Learning under non-IID data and decaying stepsize, where hard-threshold sparsification can hinder convergence. It introduces -FedHT, a stepsize-aware hard-threshold compressor with Error-Feedback that maintains compression cost and achieves convergence rates matching FedAVG for both convex and non-convex objectives, with a compression-error term scaling as . The threshold is adaptively set via using the function and , ensuring the compression adapts to learning progress. Empirical results across four non-IID benchmarks show substantial gains over Top- under equal traffic, with improvements up to on CNN-CIFAR-, demonstrating practical benefits for scalable, communication-efficient FL.

Abstract

Gradient compression can effectively alleviate communication bottlenecks in Federated Learning (FL). Contemporary state-of-the-art sparse compressors, such as Top-, exhibit high computational complexity, up to , where is the number of model parameters. The hard-threshold compressor, which simply transmits elements with absolute values higher than a fixed threshold, is thus proposed to reduce the complexity to . However, the hard-threshold compression causes accuracy degradation in FL, where the datasets are non-IID and the stepsize is decreasing for model convergence. The decaying stepsize reduces the updates and causes the compression ratio of the hard-threshold compression to drop rapidly to an aggressive ratio. At or below this ratio, the model accuracy has been observed to degrade severely. To address this, we propose -FedHT, a stepsize-aware low-cost compressor with Error-Feedback to guarantee convergence. Given that the traditional theoretical framework of FL does not consider Error-Feedback, we introduce the fundamental conversation of Error-Feedback. We prove that -FedHT has the convergence rate of ( representing total training iterations) under -strongly convex cases and under non-convex cases, \textit{same as FedAVG}. Extensive experiments demonstrate that -FedHT improves accuracy by up to over Top- under equal communication traffic on various non-IID image datasets.
Paper Structure (15 sections, 3 theorems, 24 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 3 theorems, 24 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Poor Convergence of the Hard-Threshold Compression
  4. Peeking Behind the Curtains of Poor Convergence
  5. The Stepsize-Aware Hard-Threshold Compression in FL
  6. Theoretical analysis
  7. Regularity Assumptions
  8. Convergence Rate of $\gamma$-FedHT
  9. Proof of Theorem \ref{['r-FedHT convex']}
  10. Proof of Theorem \ref{['r-FedHT non-convex']}
  11. Evaluation Experiments
  12. Experimental Settings
  13. Comparison of Model Accuracy
  14. Comparison of Communication Traffic
  15. Conclusion

Key Result

Theorem 1

Let $f: \mathbb{R}^d \rightarrow \mathbb{R}$ be $L$-smooth and $\mu$-convex. Choose $\kappa = \frac{L}{\mu}$, $b=\max \{12\frac{L}{\mu},E\}-1$ and the stepsize $\gamma_t=\frac{3}{\mu(b+t)}$. Then $\gamma$-FedHT satisfies where $B=\sum_{i=1}^n p_i^2\sigma^2 + 6L\Gamma + 8(E-1)^2 G^2 + \frac{4}{S}E^2G^2$ and $D=4d \lambda_0^2$.

Figures (4)

  • Figure 1: The global loss curves (Loss vs. Iterations) for FedAVG with hard-threshold compression (denoted as HT, left) and Top-$k$ (right) on different tasks (top to bottom). $\lambda_0$ is $\frac{\sqrt{2}}{2}$ (as well as $\frac{\sqrt{2}}{10}$) in Logistic@FMNIST (CNN@CIFAR-10). $k_0$ is $1\%$ for two tasks. The loss curves in (a, c) do not converge to the baseline, while those in (b, d) converge.
  • Figure 2: Compression curves (Compression ratio vs. Iterations) for HT with $\lambda=1$ under different settings in the convex (a-d) and non-convex (e-h) cases. Here, $E=1$ and $E=5$ denote frequent and infrequent communication, respectively, while Client: $10/10$ and $2/10$ indicate partial and full partition scenarios. Each subplot consistently demonstrates that in non-IID scenarios, the decaying-$\gamma$ prompts the hard-threshold compressor to engage in increasingly aggressive compression strategies as training progresses into the late stage.
  • Figure 3: Training curves (Accuracy vs. Iterations). The artificially non-IID partition strategy is $\#C=2$. On all benchmarks, $\gamma$-FedHT outperforms hard-threshold compression (HT) and Top-$k$.
  • Figure 4: Heatmaps of the accuracy difference ($\%$) for different types of stepsizes (denoted as $\gamma$), compression ratios ($k$), worker size ($n$) on different tasks. The accuracy difference in (a, b, d, e) is the final accuracy of $\gamma$-FedHT minus that of Top-$k$, and the difference in (c, f) is $\gamma$-FedHT-Q minus STC. In all combinations, $\gamma$-FedHT (as well as $\gamma$-FedHT-Q) is superior to Top-$k$ (STC).

Theorems & Definitions (9)

  • Theorem 1: $\mu$-strongly convex Convergence rate of $\gamma$-FedHT
  • Remark 1
  • Remark 2
  • Remark 3
  • Theorem 2: Non-Convex Convergence rate of $\gamma$-FedHT
  • Remark 4
  • Remark 5
  • Remark 6
  • Lemma 1