Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads

Wei Zhao, Anand Jayarajan, Gennady Pekhimenko

TL;DR

Tally tackles the problem of GPU underutilization in production DL clusters by offering a non-intrusive GPU sharing system that preserves the performance of latency-critical tasks. It achieves this through a virtualization layer that intercepts GPU kernel launches and applies block-level scheduling via kernel slicing and preemption, guided by a profile-driven scheduler that optimizes for turnaround latency. The main contributions include kernel transformation techniques (slicing and preemption) with a unified synchronization transformation, a priority-aware scheduling algorithm, and a transparent profiling mechanism that selects configurations at runtime. Empirical results show only $7.2\%$ average overhead on the $99^{th}$-percentile latency for high-priority inference tasks and about $80\%$ of TGS’s system throughput, demonstrating strong performance isolation with competitive efficiency for diverse DL workloads.

Abstract

GPU underutilization is a significant concern in many production deep learning clusters, leading to prolonged job queues and increased operational expenses. A promising solution to this inefficiency is GPU sharing, which improves resource utilization by allowing multiple workloads to execute concurrently on a single GPU. However, deploying GPU sharing in production settings faces critical obstacles due to the limitations of existing mechanisms, including high integration costs, inadequate performance isolation, and limited application compatibility. To address these issues, we introduce \emph{Tally}, a non-intrusive GPU sharing mechanism that provides robust performance isolation and comprehensive workload compatibility. The key to Tally's robust performance isolation capability lies in its fine-grained thread-block-level GPU kernel scheduling strategy, which allows the system to effectively mitigate interference caused by workload co-execution. We evaluate Tally on a diverse range of workloads and show that it incurs an average overhead of only $7.2\%$ on the $99^{th}$-percentile latency of high-priority inference tasks when executed concurrently with best-effort training workloads, compared to $188.9\%$ overhead exhibited by the state-of-the-art GPU sharing systems like TGS, while achieving over $80\%$ of TGS's system throughput.

Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads

TL;DR

Tally tackles the problem of GPU underutilization in production DL clusters by offering a non-intrusive GPU sharing system that preserves the performance of latency-critical tasks. It achieves this through a virtualization layer that intercepts GPU kernel launches and applies block-level scheduling via kernel slicing and preemption, guided by a profile-driven scheduler that optimizes for turnaround latency. The main contributions include kernel transformation techniques (slicing and preemption) with a unified synchronization transformation, a priority-aware scheduling algorithm, and a transparent profiling mechanism that selects configurations at runtime. Empirical results show only average overhead on the -percentile latency for high-priority inference tasks and about of TGS’s system throughput, demonstrating strong performance isolation with competitive efficiency for diverse DL workloads.

Abstract

GPU underutilization is a significant concern in many production deep learning clusters, leading to prolonged job queues and increased operational expenses. A promising solution to this inefficiency is GPU sharing, which improves resource utilization by allowing multiple workloads to execute concurrently on a single GPU. However, deploying GPU sharing in production settings faces critical obstacles due to the limitations of existing mechanisms, including high integration costs, inadequate performance isolation, and limited application compatibility. To address these issues, we introduce \emph{Tally}, a non-intrusive GPU sharing mechanism that provides robust performance isolation and comprehensive workload compatibility. The key to Tally's robust performance isolation capability lies in its fine-grained thread-block-level GPU kernel scheduling strategy, which allows the system to effectively mitigate interference caused by workload co-execution. We evaluate Tally on a diverse range of workloads and show that it incurs an average overhead of only on the -percentile latency of high-priority inference tasks when executed concurrently with best-effort training workloads, compared to overhead exhibited by the state-of-the-art GPU sharing systems like TGS, while achieving over of TGS's system throughput.

Paper Structure

This paper contains 31 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. GPU SHARING IN THE WILD
  4. TALLY: SYSTEM DESIGN
  5. Kernel Transformations
  6. Priority-aware Scheduling
  7. Non-intrusive Virtualization Layer
  8. EVALUATION
  9. Methodology
  10. End-to-end Results
  11. Traffic Load Sensitivity Analysis
  12. Scalability with Number of Workloads
  13. Performance Decomposition
  14. Turnaround Latency Analysis
  15. Overhead Analysis
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Overview of the GPU programming model.
  • Figure 2: Tally architecture.
  • Figure 3: Kernel transformations: (a) slicing and preemption transformations (b) unified synchronization transformation.
  • Figure 4: Tally's priority-aware scheduling algorithm.
  • Figure 5: Analysis of 99th-percentile latency and system throughput across different high-priority inference and best-effort training workload combinations under various GPU sharing mechanisms.
  • ...and 2 more figures