Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Layer Hierarchical Federated Learning with Quantization

Seyed Mohammad Azimi-Abarghouyi, Carlo Fischione

TL;DR

This work extends federated learning to arbitrary multi-layer hierarchies by introducing QMLHFL, a quantization-aware nested-aggregation framework. It provides a general convergence analysis showing how the convergence speed scales with the product of intra-layer iterations and how quantization and network depth affect the bound, and it formulates a geometric-programming approach to optimally allocate intra-layer iterations under deadlines. The method is validated on MNIST and CIFAR-10, demonstrating faster run-time to convergence and higher accuracy when using optimized layer-wise iterations and quantization, especially at greater depths. The results highlight the practical impact of depth-enabled, communication-efficient FL for large-scale, heterogeneous networks, enabling scalable deployment across edge-to-cloud infrastructures.

Abstract

Almost all existing hierarchical federated learning (FL) models are limited to two aggregation layers, restricting scalability and flexibility in complex, large-scale networks. In this work, we propose a Multi-Layer Hierarchical Federated Learning framework (QMLHFL), which appears to be the first study that generalizes hierarchical FL to arbitrary numbers of layers and network architectures through nested aggregation, while employing a layer-specific quantization scheme to meet communication constraints. We develop a comprehensive convergence analysis for QMLHFL and derive a general convergence condition and rate that reveal the effects of key factors, including quantization parameters, hierarchical architecture, and intra-layer iteration counts. Furthermore, we determine the optimal number of intra-layer iterations to maximize the convergence rate while meeting a deadline constraint that accounts for both communication and computation times. Our results show that QMLHFL consistently achieves high learning accuracy, even under high data heterogeneity, and delivers notably improved performance when optimized, compared to using randomly selected values.

Multi-Layer Hierarchical Federated Learning with Quantization

TL;DR

This work extends federated learning to arbitrary multi-layer hierarchies by introducing QMLHFL, a quantization-aware nested-aggregation framework. It provides a general convergence analysis showing how the convergence speed scales with the product of intra-layer iterations and how quantization and network depth affect the bound, and it formulates a geometric-programming approach to optimally allocate intra-layer iterations under deadlines. The method is validated on MNIST and CIFAR-10, demonstrating faster run-time to convergence and higher accuracy when using optimized layer-wise iterations and quantization, especially at greater depths. The results highlight the practical impact of depth-enabled, communication-efficient FL for large-scale, heterogeneous networks, enabling scalable deployment across edge-to-cloud infrastructures.

Abstract

Almost all existing hierarchical federated learning (FL) models are limited to two aggregation layers, restricting scalability and flexibility in complex, large-scale networks. In this work, we propose a Multi-Layer Hierarchical Federated Learning framework (QMLHFL), which appears to be the first study that generalizes hierarchical FL to arbitrary numbers of layers and network architectures through nested aggregation, while employing a layer-specific quantization scheme to meet communication constraints. We develop a comprehensive convergence analysis for QMLHFL and derive a general convergence condition and rate that reveal the effects of key factors, including quantization parameters, hierarchical architecture, and intra-layer iteration counts. Furthermore, we determine the optimal number of intra-layer iterations to maximize the convergence rate while meeting a deadline constraint that accounts for both communication and computation times. Our results show that QMLHFL consistently achieves high learning accuracy, even under high data heterogeneity, and delivers notably improved performance when optimized, compared to using randomly selected values.
Paper Structure (18 sections, 3 theorems, 72 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 18 sections, 3 theorems, 72 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Why Multi-Layer Hierarchical FL is Necessary
  3. Related Works
  4. Contributions
  5. System Model
  6. Multi-Layer Setup
  7. Learning Problem
  8. Multi-Layer Hierarchical Federated Learning
  9. Algorithm
  10. Quantization
  11. Convergence Analysis
  12. Learning Optimization
  13. Experimental Results
  14. Data Distribution and Hierarchical Setups
  15. MNIST Evaluation with MLP
  16. ...and 3 more sections

Key Result

Theorem 1

Assume that the learning rate $\mu$ satisfies the following condition: where ${\cal A}_n$ is a recursive function defined as with The $\max$ operation is conducted with respect to the device counts. Then, the convergence rate of QMLHFL after $T$ global iterations is bounded as

Figures (5)

  • Figure 1: An example of a 3-layer hierarchical system where the cloud server is represented by a green node, edge servers at layer 2 are shown as orange nodes, edge servers at layer 1 as red nodes, and devices as blue nodes. Node sizes increase with hierarchy level. In this system, $N_\text{tot} = 11$, $C_1 = 4$, $C_2 = 2$, $|{\cal C}_1^{1}| = 3$, $|{\cal C}_1^{2}| = 1$, $|{\cal C}_1^{3}| = 2$, $|{\cal C}_1^{4}| = 5$, $|{\cal C}_2^{1}| = 4$, and $|{\cal C}_2^{2}| = 7$.
  • Figure 2: Test accuracy of the 3-layer QMLHFL under the data heterogeneity cases: (a) Case 1, (b) Case 2, and (c) Case 3.
  • Figure 3: Test accuracy of the 4-layer QMLHFL under the data heterogeneity cases: (a) Case 1, (b) Case 2, and (c) Case 3.
  • Figure 4: Performance of the 4-layer QMLHFL under the data heterogeneity cases: (a) training loss for Case 1, (b) training loss for Case 2, and (c) test accuracy for Case 3.
  • Figure 5: Test accuracy as a function of run-time

Theorems & Definitions (10)

  • Theorem 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Remark 7
  • Corollary 1
  • Corollary 2