Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated learning compression designed for lightweight communications

Lucas Grativol Ribeiro, Mathieu Leonardon, Guillaume Muller, Virginie Fresse, Matthieu Arzel

TL;DR

This paper investigates the impact of compression techniques on FL for a typical image classification task and demonstrates that a straightforward method can compresses messages up to 50% while having less than 1% of accuracy loss, competing with state-of-the-art techniques.

Abstract

Federated Learning (FL) is a promising distributed method for edge-level machine learning, particularly for privacysensitive applications such as those in military and medical domains, where client data cannot be shared or transferred to a cloud computing server. In many use-cases, communication cost is a major challenge in FL due to its natural intensive network usage. Client devices, such as smartphones or Internet of Things (IoT) nodes, have limited resources in terms of energy, computation, and memory. To address these hardware constraints, lightweight models and compression techniques such as pruning and quantization are commonly adopted in centralised paradigms. In this paper, we investigate the impact of compression techniques on FL for a typical image classification task. Going further, we demonstrate that a straightforward method can compresses messages up to 50% while having less than 1% of accuracy loss, competing with state-of-the-art techniques.

Federated learning compression designed for lightweight communications

TL;DR

This paper investigates the impact of compression techniques on FL for a typical image classification task and demonstrates that a straightforward method can compresses messages up to 50% while having less than 1% of accuracy loss, competing with state-of-the-art techniques.

Abstract

Federated Learning (FL) is a promising distributed method for edge-level machine learning, particularly for privacysensitive applications such as those in military and medical domains, where client data cannot be shared or transferred to a cloud computing server. In many use-cases, communication cost is a major challenge in FL due to its natural intensive network usage. Client devices, such as smartphones or Internet of Things (IoT) nodes, have limited resources in terms of energy, computation, and memory. To address these hardware constraints, lightweight models and compression techniques such as pruning and quantization are commonly adopted in centralised paradigms. In this paper, we investigate the impact of compression techniques on FL for a typical image classification task. Going further, we demonstrate that a straightforward method can compresses messages up to 50% while having less than 1% of accuracy loss, competing with state-of-the-art techniques.
Paper Structure (9 sections, 3 figures, 2 tables)

This paper contains 9 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Overview of Federated Learning
  4. Model Compression
  5. Magnitude Pruning for Federated Learning
  6. Experiments
  7. Exploring Magnitude Pruning
  8. Compressing more with Quantization
  9. Conclusion

Figures (3)

  • Figure 1: The pipeline of our study. We propose a simple way to insert the pruning technique as extra step before communicating training results.
  • Figure 2: Pruning effect on the accuracy in function of the pruning rate, where the rate represents the % of total parameters pruned, for 1 and 10 clients epochs.
  • Figure 3: Accuracy evolution comparison between baseline (32-bit FP), 1-bit, 4-bit and 8-bit, for 1 and 10 clients epochs.