Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging

Michail Theologitis, Georgios Frangias, Georgios Anestis, Vasilis Samoladas, Antonios Deligiannakis

TL;DR

Federated Dynamic Averaging is proposed, a communication-efficient DDL strategy that dynamically triggers synchronization based on the value of the model variance, which reduces communication cost by orders of magnitude, compared to both traditional and cutting-edge communication-efficient algorithms.

Abstract

The ever-growing volume and decentralized nature of data, coupled with the need to harness it and extract knowledge, have led to the extensive use of distributed deep learning (DDL) techniques for training. These techniques rely on local training performed at distributed nodes using locally collected data, followed by a periodic synchronization process that combines these models to create a unified global model. However, the frequent synchronization of deep learning models, encompassing millions to many billions of parameters, creates a communication bottleneck, severely hindering scalability. Worse yet, DDL algorithms typically waste valuable bandwidth and render themselves less practical in bandwidth-constrained federated settings by relying on overly simplistic, periodic, and rigid synchronization schedules. These inefficiencies make the training process increasingly impractical as they demand excessive time for data communication. To address these shortcomings, we propose Federated Dynamic Averaging (FDA), a communication-efficient DDL strategy that dynamically triggers synchronization based on the value of the model variance. In essence, the costly synchronization step is triggered only if the local models -- initialized from a common global model after each synchronization -- have significantly diverged. This decision is facilitated by the transmission of a small local state from each distributed node. Through extensive experiments across a wide range of learning tasks we demonstrate that FDA reduces communication cost by orders of magnitude, compared to both traditional and cutting-edge communication-efficient algorithms. Additionally, we show that FDA maintains robust performance across diverse data heterogeneity settings.

Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging

TL;DR

Federated Dynamic Averaging is proposed, a communication-efficient DDL strategy that dynamically triggers synchronization based on the value of the model variance, which reduces communication cost by orders of magnitude, compared to both traditional and cutting-edge communication-efficient algorithms.

Abstract

The ever-growing volume and decentralized nature of data, coupled with the need to harness it and extract knowledge, have led to the extensive use of distributed deep learning (DDL) techniques for training. These techniques rely on local training performed at distributed nodes using locally collected data, followed by a periodic synchronization process that combines these models to create a unified global model. However, the frequent synchronization of deep learning models, encompassing millions to many billions of parameters, creates a communication bottleneck, severely hindering scalability. Worse yet, DDL algorithms typically waste valuable bandwidth and render themselves less practical in bandwidth-constrained federated settings by relying on overly simplistic, periodic, and rigid synchronization schedules. These inefficiencies make the training process increasingly impractical as they demand excessive time for data communication. To address these shortcomings, we propose Federated Dynamic Averaging (FDA), a communication-efficient DDL strategy that dynamically triggers synchronization based on the value of the model variance. In essence, the costly synchronization step is triggered only if the local models -- initialized from a common global model after each synchronization -- have significantly diverged. This decision is facilitated by the transmission of a small local state from each distributed node. Through extensive experiments across a wide range of learning tasks we demonstrate that FDA reduces communication cost by orders of magnitude, compared to both traditional and cutting-edge communication-efficient algorithms. Additionally, we show that FDA maintains robust performance across diverse data heterogeneity settings.
Paper Structure (11 sections, 2 theorems, 17 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 2 theorems, 17 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Dynamic Averaging
  4. SketchFDA: Sketch-based Estimation
  5. LinearFDA: Linear Approximation
  6. Discussion
  7. Experiments
  8. Setup
  9. Main Findings
  10. Results
  11. Conclusions and Future Work

Key Result

Theorem 3.1

Let $l = \mathcal{O}(\log \frac{1}{\delta})$ and $m = \mathcal{O}(\frac{1}{ \epsilon^2})$. Define the local state as and the approximation function as Then, the condition $H\large(\overline{ \mathbf{S} }_t\large) \leq \Theta$ implies $\operatorname{\mathcal{V}\!\text{ar}}\left( \mathbf{w} _t\right) \leq \Theta$ with probability at least ($1-\delta$).

Figures (13)

  • Figure 1: FDA. The local training step is followed by the computation of a local state by all worker-nodes. Then, the (small in size) local states are aggregated. Based on the aggregated result, all workers estimate if synchronization is required. In most cases, the expensive synchronization step of the models is avoided and local training continues
  • Figure 2: SketchFDA & LinearFDA: Local State structure.
  • Figure 3: LeNet-5 on MNIST. At Non-IID: Label "0", the samples of Label "0" are assigned to few workers. At Non-IID: 60%, 60% of the dataset is sorted and allocated to workers, causing some workers to receive many samples from the same label
  • Figure 4: VGG16* on MNIST
  • Figure 5: DenseNet121 on CIFAR-10
  • ...and 8 more figures

Theorems & Definitions (2)

  • Theorem 3.1
  • Theorem 3.2