Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale

TL;DR

This work tackles the challenge of meeting strict SLOs for high-priority tasks while maximizing GPU utilization in the context of coarse-grained GPU sharing on NVIDIA GPUs, where fine-grained scheduling is not exposed. It introduces Hummingbird, a three-part system consisting of a Kernel Splitter, a Runtime Scheduler, and a memory manager with NVLink-based HUVM extension, to achieve microsecond-scale preemption and efficient bubble harvesting, guarded by a kernel-tick scheduling policy. Empirically, it delivers up to $9.7\times$ and $3.5\times$ improvements in $P_{99}$ latency attainment over spatial and temporal baselines, respectively, with less than $1\%$ overhead to high-priority latency and a $2.4\times$ boost in low-priority throughput, across single- and multi-GPU deployments and memory-intensive scenarios. The approach, including PTX-based kernel transformation and host-assisted bubble detection, generalizes to multiple GPU generations and frameworks, offering a practical pathway to high utilization without sacrificing SLO guarantees in real-world data-center clusters.

Abstract

Existing GPU-sharing techniques, including spatial and temporal sharing, aim to improve utilization but face challenges in simultaneously ensuring SLO adherence and maximizing efficiency due to the lack of fine-grained task scheduling on closed-source GPUs. This paper presents Hummingbird, an SLO-oriented GPU scheduling system that overcomes these challenges by enabling microsecond-scale preemption on closed-source GPUs while effectively harvesting idle GPU time slices. Comprehensive evaluations across diverse GPU architectures reveal that Hummingbird improves the SLO attainment of high-priority tasks by 9.7x and 3.5x compared to the state-of-the-art spatial and temporal-sharing approaches. When compared to executing exclusively, the SLO attainment of the high-priority task, collocating with low-priority tasks on Hummingbird, only drops by less than 1%. Meanwhile, the throughput of the low-priority task outperforms the state-of-the-art temporal-sharing approaches by 2.4x. Hummingbird demonstrates significant effectiveness in ensuring the SLO while enhancing GPU utilization.

Figures (14)

• Figure 1: Overview of GPU-sharing techniques. The X-axis represents GPU utilization, while the Y-axis (SLO attainment) represents the proportion of high-priority requests that meet the SLO. The SLO is defined as the 99th percentile (P99) latency for exclusive execution as in previous works Autothrottleconserve.
• Figure 2: (a) The proportion of small bubbles when processing a request; (b) The distribution of small bubble time (µs).
• Figure 3: (a) The distribution of kernel execution time (µs); (b) The CDF of the thread block execution time (µs).
• Figure 4: An illustration of GPU task scheduling across two co-run schemes: (a) Orion, (b) REEF.
• Figure 5: (a) SLO attainment of high-priority task (gray lines) and throughput of low-priority tasks (bars-TPS); (b) Boxplot of the time per output token (TPOT) of high-priority tasks.