FIKIT: Priority-Based Real-time GPU Multi-tasking Scheduling with Kernel Identification

TL;DR

FIKIT tackles priority-aware intra-GPU multitasking in cloud environments by exploiting inter-kernel idle time to run low-priority kernels during high-priority gaps. It deploys a kernel-level scheduling framework with offline profiling and a kernel identification mechanism enabled by recompiling ML libraries with -rdynamic, plus a central scheduler that uses up to 10 priority queues and a BestPrioFit policy to fill idle intervals. The approach separates measurement from real-time sharing to minimize scheduling overhead, achieving high-priority speedups from $1.32\times$ to $16.41\times$, with over half the cases exceeding $3.4\times$, and maintains stable low-priority JCT under preemption, while keeping runtime overhead under 5%. However, online kernel measurement incurs substantial JCT increase during the profiling phase, motivating the two-stage design. Overall, FIKIT demonstrates practical, non-intrusive improvements for cloud GPU multitasking, enabling efficient, priority-driven utilization without requiring open-source driver modifications.

Abstract

Highly parallelized workloads like machine learning training, inferences and general HPC tasks are greatly accelerated using GPU devices. In a cloud computing cluster, serving a GPU's computation power through multi-tasks sharing is highly demanded since there are always more task requests than the number of GPU available. Existing GPU sharing solutions focus on reducing task-level waiting time or task-level switching costs when multiple jobs competing for a single GPU. Non-stopped computation requests come with different priorities, having non-symmetric impact on QoS for sharing a GPU device. Existing work missed the kernel-level optimization opportunity brought by this setting. To address this problem, we present a novel kernel-level scheduling strategy called FIKIT: Filling Inter-kernel Idle Time. FIKIT incorporates task-level priority information, fine-grained kernel identification, and kernel measurement, allowing low priorities task's execution during high priority task's inter-kernel idle time. Thereby, filling the GPU's device runtime fully, and reduce overall GPU sharing impact to cloud services. Across a set of ML models, the FIKIT based inference system accelerated high priority tasks by 1.32 to 16.41 times compared to the JCT in GPU sharing mode, and more than half of the cases are accelerated by more than 3.4 times. Alternatively, under preemptive sharing, the low-priority tasks have a comparable to default GPU sharing mode JCT, with a 0.86 to 1 times ratio. We further limit the kernel measurement and runtime fine-grained kernel scheduling overhead to less than 5%.

Figures (21)

• Figure 1: A GPU task has gaps between kernels.
• Figure 2: Using an idealized two tasks sharing case study, we demonstrated how FIKIT works compared to Nvidia GPU sharing and exclusive scheduling.
• Figure 3: A service's execution is divided into two phases: measurement and FIKIT execution phase.
• Figure 4: The construction of a Kernel ID.
• Figure 5: There can be kernels of different execution duration but with the same kernel ID.