Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Federated Learning with SignSGD: A Highly Communication-Efficient Approach

Amirreza Kazemi, Seyed Mohammad Azimi-Abarghouyi, Gabor Fodor, Carlo Fischione

TL;DR

This work tackles the communication bottlenecks of hierarchical federated learning by introducing HierSignSGD, a sign-based framework where devices transmit only gradient signs, edge servers perform majority-vote aggregation, and the cloud periodically averages edge models, with optional downlink quantization. The authors formulate a two-layer HFL objective and provide a nonconvex convergence analysis that quantifies how biased sign compression, two-level aggregation, and inter-cluster heterogeneity affect convergence, showing a sublinear $\mathcal{O}(1/\sqrt{T_G})$ rate under IID inter-edge data. Empirical results on EMNIST-digits and Fashion-MNIST demonstrate competitive accuracy relative to full-precision hierarchical SGD while drastically reducing communication, and reveal robustness to aggressive downlink sparsification and an optimal edge-device iteration count. The framework offers practical scalability for wireless and IoT deployments, with insights on clustering and aggregation timings that inform real-world system design.

Abstract

Hierarchical federated learning (HFL) has emerged as a key architecture for large-scale wireless and Internet of Things systems, where devices communicate with nearby edge servers before reaching the cloud. In these environments, uplink bandwidth and latency impose strict communication limits, thereby making aggressive gradient compression essential. One-bit methods such as sign-based stochastic gradient descent (SignSGD) offer an attractive solution in flat federated settings, but existing theory and algorithms do not naturally extend to hierarchical settings. In particular, the interaction between majority-vote aggregation at the edge layer and model aggregation at the cloud layer, and its impact on end-to-end performance, remains unknown. To bridge this gap, we propose a highly communication-efficient sign-based HFL framework and develop its corresponding formulation for nonconvex learning, where devices send only signed stochastic gradients, edge servers combine them through majority-vote, and the cloud periodically averages the obtained edge models, while utilizing downlink quantization to broadcast the global model. We introduce the resulting scalable HFL algorithm, HierSignSGD, and provide the convergence analysis for SignSGD in a hierarchical setting. Our core technical contribution is a characterization of how biased sign compression, two-level aggregation intervals, and inter-cluster heterogeneity collectively affect convergence. Numerical experiments under homogeneous and heterogeneous data splits show that HierSignSGD, despite employing extreme compression, achieves accuracy comparable to or better than full-precision stochastic gradient descent while reducing communication cost in the process, and remains robust under aggressive downlink sparsification.

Hierarchical Federated Learning with SignSGD: A Highly Communication-Efficient Approach

TL;DR

This work tackles the communication bottlenecks of hierarchical federated learning by introducing HierSignSGD, a sign-based framework where devices transmit only gradient signs, edge servers perform majority-vote aggregation, and the cloud periodically averages edge models, with optional downlink quantization. The authors formulate a two-layer HFL objective and provide a nonconvex convergence analysis that quantifies how biased sign compression, two-level aggregation, and inter-cluster heterogeneity affect convergence, showing a sublinear rate under IID inter-edge data. Empirical results on EMNIST-digits and Fashion-MNIST demonstrate competitive accuracy relative to full-precision hierarchical SGD while drastically reducing communication, and reveal robustness to aggressive downlink sparsification and an optimal edge-device iteration count. The framework offers practical scalability for wireless and IoT deployments, with insights on clustering and aggregation timings that inform real-world system design.

Abstract

Hierarchical federated learning (HFL) has emerged as a key architecture for large-scale wireless and Internet of Things systems, where devices communicate with nearby edge servers before reaching the cloud. In these environments, uplink bandwidth and latency impose strict communication limits, thereby making aggressive gradient compression essential. One-bit methods such as sign-based stochastic gradient descent (SignSGD) offer an attractive solution in flat federated settings, but existing theory and algorithms do not naturally extend to hierarchical settings. In particular, the interaction between majority-vote aggregation at the edge layer and model aggregation at the cloud layer, and its impact on end-to-end performance, remains unknown. To bridge this gap, we propose a highly communication-efficient sign-based HFL framework and develop its corresponding formulation for nonconvex learning, where devices send only signed stochastic gradients, edge servers combine them through majority-vote, and the cloud periodically averages the obtained edge models, while utilizing downlink quantization to broadcast the global model. We introduce the resulting scalable HFL algorithm, HierSignSGD, and provide the convergence analysis for SignSGD in a hierarchical setting. Our core technical contribution is a characterization of how biased sign compression, two-level aggregation intervals, and inter-cluster heterogeneity collectively affect convergence. Numerical experiments under homogeneous and heterogeneous data splits show that HierSignSGD, despite employing extreme compression, achieves accuracy comparable to or better than full-precision stochastic gradient descent while reducing communication cost in the process, and remains robust under aggressive downlink sparsification.
Paper Structure (17 sections, 4 theorems, 63 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 4 theorems, 63 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works and Challenges
  3. The HFL Problem
  4. The $\mathtt{HierSignSGD}$ Algorithm
  5. Pseudocode
  6. Convergence Analysis
  7. Extension to Downlink Quantization
  8. Simulations
  9. Setup
  10. Learning Accuracy
  11. Effect of $T_E$
  12. Cluster Reconfiguration
  13. Effect of Downlink Quantization
  14. Conclusion
  15. Majority-Vote Error Analysis
  16. ...and 2 more sections

Key Result

Theorem 1

Consider running Algorithm alg:hier-SgnSGD with single-device clusters for $T_G$ global iterations and $T_E$ local iterations, using step-size $\mu$ and batch-size $B$. Under assumptions A1–A4, the following performance bound holds: where

Figures (6)

  • Figure 1: SignSGD-based implementation for a HFL scenario; devices send gradient signs to their edge servers, the servers broadcast majority-vote results back; after several rounds, the edge servers forward the model parameters to the cloud for final aggregation.
  • Figure 2: EMNIST-digits proportions per edge server under dirichlet non-IID partition $(\alpha = 0.3)$
  • Figure 3: Test accuracy comparison between $\mathtt{HierSignSGD}$ and $\mathtt{HierSGD}$ on EMNIST-digits and Fashion-MNIST. We use a batch-size of $B=400$. Step-size $\mu$: EMNIST---1 ($\mathtt{HierSGD}$), $5\times10^{-3}$ ($\mathtt{HierSignSGD}$); Fashion-MNIST---0.1 ($\mathtt{HierSGD}$), $7\times10^{-4}$ ($\mathtt{HierSignSGD}$).
  • Figure 4: Effect of $T_E$ on global training loss for $\mathtt{HierSignSGD}$.
  • Figure 5: Training sensitivity to different clusterings of $M\times Q=48$ IID devices.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Theorem 1
  • proof
  • Corollary 2
  • Theorem 3
  • proof
  • Theorem 4
  • proof