Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression

Xiaoxin Su, Yipeng Zhou, Laizhong Cui, Song Guo

TL;DR

This work tackles the memory and communication bottlenecks of in-network federated learning coordinated by programmable switches. It introduces FediAC, a two-phase consensus-based compression where Phase 1 uses 0-1 voting to identify globally significant updates and Phase 2 uploads quantized updates aligned to this consensus, enabling memory-efficient, pipelined aggregation. The authors prove convergence under a biased yet controlled compression bound and demonstrate that FediAC achieves higher accuracy (up to ~7.7 percentage points) and substantially lowers communication traffic (up to ~69% reduction) across CIFAR-10/100 and FEMNIST compared to state-of-the-art baselines. They also provide practical guidance on tuning parameters and show robustness to non-IID data and system scale, with a clear path toward extension to networks with multiple collaborative PS devices.

Abstract

Recently, federated learning (FL) has gained momentum because of its capability in preserving data privacy. To conduct model training by FL, multiple clients exchange model updates with a parameter server via Internet. To accelerate the communication speed, it has been explored to deploy a programmable switch (PS) in lieu of the parameter server to coordinate clients. The challenge to deploy the PS in FL lies in its scarce memory space, prohibiting running memory consuming aggregation algorithms on the PS. To overcome this challenge, we propose Federated Learning in-network Aggregation with Compression (FediAC) algorithm, consisting of two phases: client voting and model aggregating. In the former phase, clients report their significant model update indices to the PS to estimate global significant model updates. In the latter phase, clients upload global significant model updates to the PS for aggregation. FediAC consumes much less memory space and communication traffic than existing works because the first phase can guarantee consensus compression across clients. The PS easily aligns model update indices to swiftly complete aggregation in the second phase. Finally, we conduct extensive experiments by using public datasets to demonstrate that FediAC remarkably surpasses the state-of-the-art baselines in terms of model accuracy and communication traffic.

Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression

TL;DR

This work tackles the memory and communication bottlenecks of in-network federated learning coordinated by programmable switches. It introduces FediAC, a two-phase consensus-based compression where Phase 1 uses 0-1 voting to identify globally significant updates and Phase 2 uploads quantized updates aligned to this consensus, enabling memory-efficient, pipelined aggregation. The authors prove convergence under a biased yet controlled compression bound and demonstrate that FediAC achieves higher accuracy (up to ~7.7 percentage points) and substantially lowers communication traffic (up to ~69% reduction) across CIFAR-10/100 and FEMNIST compared to state-of-the-art baselines. They also provide practical guidance on tuning parameters and show robustness to non-IID data and system scale, with a clear path toward extension to networks with multiple collaborative PS devices.

Abstract

Recently, federated learning (FL) has gained momentum because of its capability in preserving data privacy. To conduct model training by FL, multiple clients exchange model updates with a parameter server via Internet. To accelerate the communication speed, it has been explored to deploy a programmable switch (PS) in lieu of the parameter server to coordinate clients. The challenge to deploy the PS in FL lies in its scarce memory space, prohibiting running memory consuming aggregation algorithms on the PS. To overcome this challenge, we propose Federated Learning in-network Aggregation with Compression (FediAC) algorithm, consisting of two phases: client voting and model aggregating. In the former phase, clients report their significant model update indices to the PS to estimate global significant model updates. In the latter phase, clients upload global significant model updates to the PS for aggregation. FediAC consumes much less memory space and communication traffic than existing works because the first phase can guarantee consensus compression across clients. The PS easily aligns model update indices to swiftly complete aggregation in the second phase. Finally, we conduct extensive experiments by using public datasets to demonstrate that FediAC remarkably surpasses the state-of-the-art baselines in terms of model accuracy and communication traffic.
Paper Structure (21 sections, 3 theorems, 8 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 21 sections, 3 theorems, 8 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Traditional FL
  5. A Motivation Example
  6. Algorithm Design and Analysis
  7. FediAC Algorithm Design
  8. Compression Error Analysis
  9. Convergence Analysis
  10. Implementation of FediAC
  11. Performance Evaluation
  12. Experimental Settings
  13. Datasets and Models
  14. System Settings
  15. Baselines
  16. ...and 6 more sections

Key Result

Proposition 1

In FediAC, each client votes a model update in $\mathbf{U}^i_t$ with the odds proportional to its magnitude. When a model update is voted by at least $a$ clients, the model update is scaled up by $f$ and quantized to an integer of $b$ bits by EQ:QuantizationEq. Then, the compression error of FediAC where $f=\frac{2^{b-1}-N}{Nm}$ and $m$ is the maximum value of model update magnitudes. Recall that

Figures (4)

  • Figure 1: The training process of FediAC.
  • Figure 2: Comparing model accuracy with difference datasets and data distributions with a high performance (top) PS and a low performance (bottom) PS.
  • Figure 3: Comparing model accuracy of FediAC and libra by varying the non-IID degree in CIFAR-10 with the high performance PS (left) and the low performance PS (right).
  • Figure 4: Comparing model accuracy trained by FediAC using different hyperparameter $a$ under different system scales with IID (left) and non-IID (right) data distributions in CIFAR-10.

Theorems & Definitions (5)

  • Definition 1
  • Proposition 1
  • Corollary 1
  • Definition 2
  • Theorem 1