An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs

TL;DR

The paper tackles inefficiencies in MIG-based GPU sharing caused by contention and fragmentation from fixed MIG profiles. It introduces an online fragmentation-aware scheduler that combines conditional load balancing, dynamic partitioning, and job migration to optimize placement and reconfiguration on a single node. Through experimental evaluation on a 4-GPU node, the approach achieves up to 35% makespan reduction when all techniques are enabled, driven by reduced contention and better GPU configurability. The work provides a practical framework for deploying MIG-enabled, multi-tenant GPU workloads with improved utilization and responsiveness.

Abstract

Modern GPU workloads increasingly demand efficient resource sharing, as many jobs do not require the full capacity of a GPU. Among sharing techniques, NVIDIA's Multi-Instance GPU (MIG) offers strong resource isolation by enabling hardware-level GPU partitioning. However, leveraging MIG effectively introduces new challenges. First, resource contention persists due to shared components such as PCIe bandwidth. Second, GPU fragmentation becomes a critical issue, which is different from prior fine-grained GPU sharing work due to MIG's limited number of valid MIG configurations. Fragmentation arises not only from spatial discontinuity but also from rigid profile placement constraints, especially after job arrivals and terminations. To address these issues, we propose an online scheduling framework that integrates conditional load balancing, dynamic partitioning, and job migration. Our approach dynamically adapts job placement to minimize contention and reorganizes GPU allocations to combat both internal and external fragmentation. Experimental results show that our method significantly improves system efficiency. When all techniques are applied, the makespan improves by up to 35%.

Figures (10)

• Figure 1: GPU 1 has a continuous 4g space but cannot serve a 4g MIG instance due to external fragmentation.
• Figure 2: External fragment occurs when jobs are finished.
• Figure 3: Scheduling upon job arrival. The conditional load-balancing scheduler prefers the Lazy GPU (the GPU with utilization lower than the load-balancing threshold) and the placement with the least fragmentation.
• Figure 4: Migration upon job departure from target GPU. The job migration planner formulates an intra-GPU migration or an inter-GPU migration plan to balance the workload across GPUs and minimize GPU fragmentation.
• Figure 5: Time per output token of different models under different numbers of concurrent tasks.