Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study

Alexander Erben, Ruben Mayer, Hans-Arno Jacobsen

TL;DR

The paper tackles the challenge of cost-efficiently training deep learning models using spot instances across clouds and continents. It leverages the Hivemind framework to enable decentralized, data-parallel training on interruptible hardware and introduces a granularity metric to predict geo-distributed scalability. Through extensive experiments on CV, NLP, and ASR tasks across geo-distributed, multi-cloud, and hybrid-cloud configurations, it demonstrates meaningful speedups for small- to mid-sized models (up to ≈4.37x with 8 GPUs) while highlighting egress costs and intercontinental latency as key cost drivers. The results offer practical guidelines and metrics for practitioners to optimize cost-per-throughput when using spot pricing across multiple clouds and regions, showing that geo-distributed spot training can be a viable and cost-effective alternative to centralized on-demand or high-end single-node offerings.

Abstract

This paper aims to answer the question: Can deep learning models be cost-efficiently trained on a global market of spot VMs spanning different data centers and cloud providers? To provide guidance, we extensively evaluate the cost and throughput implications of training in different zones, continents, and clouds for representative CV, NLP, and ASR models. To expand the current training options further, we compare the scalability potential for hybrid-cloud scenarios by adding cloud resources to on-premise hardware to improve training throughput. Finally, we show how leveraging spot instance pricing enables a new cost-efficient way to train models with multiple cheap VMs, trumping both more centralized and powerful hardware and even on-demand cloud offerings at competitive prices.

How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study

TL;DR

The paper tackles the challenge of cost-efficiently training deep learning models using spot instances across clouds and continents. It leverages the Hivemind framework to enable decentralized, data-parallel training on interruptible hardware and introduces a granularity metric to predict geo-distributed scalability. Through extensive experiments on CV, NLP, and ASR tasks across geo-distributed, multi-cloud, and hybrid-cloud configurations, it demonstrates meaningful speedups for small- to mid-sized models (up to ≈4.37x with 8 GPUs) while highlighting egress costs and intercontinental latency as key cost drivers. The results offer practical guidelines and metrics for practitioners to optimize cost-per-throughput when using spot pricing across multiple clouds and regions, showing that geo-distributed spot training can be a viable and cost-effective alternative to centralized on-demand or high-end single-node offerings.

Abstract

This paper aims to answer the question: Can deep learning models be cost-efficiently trained on a global market of spot VMs spanning different data centers and cloud providers? To provide guidance, we extensively evaluate the cost and throughput implications of training in different zones, continents, and clouds for representative CV, NLP, and ASR models. To expand the current training options further, we compare the scalability potential for hybrid-cloud scenarios by adding cloud resources to on-premise hardware to improve training throughput. Finally, we show how leveraging spot instance pricing enables a new cost-efficient way to train models with multiple cheap VMs, trumping both more centralized and powerful hardware and even on-demand cloud offerings at competitive prices.
Paper Structure (13 sections, 17 figures, 6 tables)

This paper contains 13 sections, 17 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Deep Learning On Spot Instances
  3. Hivemind
  4. Distributed Spot Training
  5. Model Suitability
  6. Geo-distributed Performance
  7. Multi-cloud Performance
  8. Hybrid-cloud Performance
  9. Further Insights
  10. Lessons Learned
  11. Related Work
  12. Conclusion
  13. Appendix: ASR Case Study

Figures (17)

  • Figure 1: Cost to throughput tradeoff for ConvNextLarge at different instance types. Our training setups (circled) are cheaper (8xT4) and faster (8xA10) than centralized offerings (DGX-2).
  • Figure 2: Hivemind penalty on normalized throughputs.
  • Figure 3: Throughput comparison between single GPU baselines and the Hivemind runs with two GPUs.
  • Figure 4: TBS vs. total training time on 2xA10s. Granularity is shown above each bar. Dotted lines separate different models.
  • Figure 5: Throughput comparison from 1 to 8 A10 GPUs.
  • ...and 12 more figures