Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ScaDLES: Scalable Deep Learning over Streaming data at the Edge

Sahil Tyagi, Martin Swany

TL;DR

ScaDLES tackles the challenge of training deep networks on streaming edge data in the presence of device heterogeneity and non-IID distributions. It introduces per-device variable mini-batches, weighted gradient aggregation, memory-aware streaming policies, data-injection to mitigate non-IID effects, and adaptive gradient compression with linear learning-rate scaling. Empirical results on ResNet152 and VGG19 with CIFAR data show up to 3.29× faster convergence and substantial buffer-size reductions, while maintaining competitive accuracy under IID and substantially improving accuracy under non-IID conditions. The approach provides a practical, bandwidth-aware pathway to scalable edge DL with improved convergence and privacy-preserving characteristics suitable for real-world edge/fog deployments.

Abstract

Distributed deep learning (DDL) training systems are designed for cloud and data-center environments that assumes homogeneous compute resources, high network bandwidth, sufficient memory and storage, as well as independent and identically distributed (IID) data across all nodes. However, these assumptions don't necessarily apply on the edge, especially when training neural networks on streaming data in an online manner. Computing on the edge suffers from both systems and statistical heterogeneity. Systems heterogeneity is attributed to differences in compute resources and bandwidth specific to each device, while statistical heterogeneity comes from unbalanced and skewed data on the edge. Different streaming-rates among devices can be another source of heterogeneity when dealing with streaming data. If the streaming rate is lower than training batch-size, device needs to wait until enough samples have streamed in before performing a single iteration of stochastic gradient descent (SGD). Thus, low-volume streams act like stragglers slowing down devices with high-volume streams in synchronous training. On the other hand, data can accumulate quickly in the buffer if the streaming rate is too high and the devices can't train at line-rate. In this paper, we introduce ScaDLES to efficiently train on streaming data at the edge in an online fashion, while also addressing the challenges of limited bandwidth and training with non-IID data. We empirically show that ScaDLES converges up to 3.29 times faster compared to conventional distributed SGD.

ScaDLES: Scalable Deep Learning over Streaming data at the Edge

TL;DR

ScaDLES tackles the challenge of training deep networks on streaming edge data in the presence of device heterogeneity and non-IID distributions. It introduces per-device variable mini-batches, weighted gradient aggregation, memory-aware streaming policies, data-injection to mitigate non-IID effects, and adaptive gradient compression with linear learning-rate scaling. Empirical results on ResNet152 and VGG19 with CIFAR data show up to 3.29× faster convergence and substantial buffer-size reductions, while maintaining competitive accuracy under IID and substantially improving accuracy under non-IID conditions. The approach provides a practical, bandwidth-aware pathway to scalable edge DL with improved convergence and privacy-preserving characteristics suitable for real-world edge/fog deployments.

Abstract

Distributed deep learning (DDL) training systems are designed for cloud and data-center environments that assumes homogeneous compute resources, high network bandwidth, sufficient memory and storage, as well as independent and identically distributed (IID) data across all nodes. However, these assumptions don't necessarily apply on the edge, especially when training neural networks on streaming data in an online manner. Computing on the edge suffers from both systems and statistical heterogeneity. Systems heterogeneity is attributed to differences in compute resources and bandwidth specific to each device, while statistical heterogeneity comes from unbalanced and skewed data on the edge. Different streaming-rates among devices can be another source of heterogeneity when dealing with streaming data. If the streaming rate is lower than training batch-size, device needs to wait until enough samples have streamed in before performing a single iteration of stochastic gradient descent (SGD). Thus, low-volume streams act like stragglers slowing down devices with high-volume streams in synchronous training. On the other hand, data can accumulate quickly in the buffer if the streaming rate is too high and the devices can't train at line-rate. In this paper, we introduce ScaDLES to efficiently train on streaming data at the edge in an online fashion, while also addressing the challenges of limited bandwidth and training with non-IID data. We empirically show that ScaDLES converges up to 3.29 times faster compared to conventional distributed SGD.
Paper Structure (21 sections, 9 equations, 9 figures, 6 tables)

This paper contains 21 sections, 9 equations, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Challenges in Streaming DL
  3. Heterogeneity in device streaming rates
  4. Data skewness in deep learning
  5. Limited Memory and storage
  6. Synchronization overhead
  7. Background and Related Work
  8. Handling unbalanced and non-IID data
  9. Dealing with limited memory
  10. Reducing communication cost
  11. scadles
  12. Evaluation
  13. Cluster setup
  14. Data, models and hyperparameters
  15. Streaming data for DDL
  16. ...and 6 more sections

Figures (9)

  • Figure 1: Streaming latency across batches when device stream-rates are sampled from different distributions.
  • Figure 2: Test accuracy for ResNet152 on CIFAR10 and VGG19 on CIFAR100 with IID and non-IID data.
  • Figure 3: GPU memory utilization in DDL.
  • Figure 4: DDL on streaming data is limited by memory/storage as well as network bandwidth.
  • Figure 5: Effective streaming rates achieved when scaling to multiple topics.
  • ...and 4 more figures