Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Incremental Aggregation in Multi-Hop Federated Learning

Sourav Mukherjee, Nasrin Razmi, Armin Dekorsy, Petar Popovski, Bho Matthiesen

TL;DR

The paper tackles federated learning over multi-hop networks where in-network incremental aggregation (IA) can dramatically cut communication, but gradient sparsification degrades IA gains. It proposes correlated sparsification methods—RE-SIA, CL-SIA, TC-SIA, and CL-TC-SIA—to preserve IA efficiency under Top-$Q$ sparsification and, when used with time-correlation (tcs), control the growth of transmitted nonzeros across hops. The authors analyze error minimization, derive cost bounds, and demonstrate via MNIST-based experiments that constant-length (CL) and time-correlated (TC) variants achieve substantial bandwidth reductions while maintaining near-IA convergence, with CL-SIA/CL-TC-SIA performing best under equal bandwidth constraints. The results indicate strong potential for efficient, scalable IA in multi-hop FL, especially in satellite constellations and related networks, and point to future work on rigorous convergence guarantees for the new schemes.

Abstract

This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA.

Sparse Incremental Aggregation in Multi-Hop Federated Learning

TL;DR

The paper tackles federated learning over multi-hop networks where in-network incremental aggregation (IA) can dramatically cut communication, but gradient sparsification degrades IA gains. It proposes correlated sparsification methods—RE-SIA, CL-SIA, TC-SIA, and CL-TC-SIA—to preserve IA efficiency under Top- sparsification and, when used with time-correlation (tcs), control the growth of transmitted nonzeros across hops. The authors analyze error minimization, derive cost bounds, and demonstrate via MNIST-based experiments that constant-length (CL) and time-correlated (TC) variants achieve substantial bandwidth reductions while maintaining near-IA convergence, with CL-SIA/CL-TC-SIA performing best under equal bandwidth constraints. The results indicate strong potential for efficient, scalable IA in multi-hop FL, especially in satellite constellations and related networks, and point to future work on rigorous convergence guarantees for the new schemes.

Abstract

This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA.
Paper Structure (13 sections, 2 theorems, 8 equations, 4 figures, 5 algorithms)

This paper contains 13 sections, 2 theorems, 8 equations, 4 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. System Model
  3. Sparse Incremental Aggregation
  4. Sparse Incremental Aggregation, Revisited
  5. An Error Minimization Perspective on Sparse IA
  6. Reduced-Error Sparse Incremental Aggregation
  7. Constant-Length Sparse Incremental Aggregation
  8. Time-Correlated Sparse IA
  9. Time-Correlated Sparse Incremental Aggregation
  10. Constant-Length Time-Correlated Sparse IA
  11. Communication Cost
  12. Numerical Evaluation
  13. Conclusions

Key Result

Proposition 1

$C(\bm\gamma_{k+1}^t, \tilde{\bm g}_k^t) = S(\tilde{\bm g}_k^t, Q)$ is strictly suboptimal with respect to opt:mh unless the sparsification supports of $\bm\gamma_{k+1}^t$ and $\tilde{\bm g}_k^t$ are identical.

Figures (4)

  • Figure 1: Multi-hop federated learning system.
  • Figure 2: Total transmitted data per global iteration for fixed $Q = 78$ with respect to the number of clients.
  • Figure 3: Test accuracy for a fixed $Q = 78$ and $K = 28$ clients.
  • Figure 4: Test accuracy for $K = 28$ clients under (approximately) equal average bandwidth usage of 98k per global iteration.

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2