Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Workload Placement on Multi-Instance GPUs

Bekir Turkkan, Pavankumar Murali, Pavithra Harsha, Rohan Arora, Gerard Vanloo, Chandra Narayanaswami

TL;DR

The paper tackles optimizing Large Language Model (LLM) inference workloads on Multi-Instance GPUs (MIGs) to minimize total GPU usage and wasted compute/memory. It introduces a framework with a placement recommender, migration planner, and executor, and presents two approaches: a global Mixed-Integer Programming (MIP) method and a fast rule-based heuristic, both addressing initial deployment, compaction, and reconfiguration under Dynamic Resource Allocation (DRA). The core contribution is the WPM MIP model and a tractable heuristic, along with preprocessing to handle partially partitioned GPUs; evaluations show up to 2.85x GPU reduction and up to 70% wastage reduction versus baselines, with competitive performance from the heuristic. The work offers a practical path toward production-ready MIG-aware scheduling, enabling non-disruptive migrations and more available GPU resources for future workloads.

Abstract

There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we investigate how to optimize the placement of Large Language Model (LLM)-based AI Inferencing workloads on GPUs. We first identify and present several use cases that are encountered in practice that require workloads to be efficiently placed or migrated to other GPUs to make room for incoming workloads. The overarching goal is to use as few GPUs as possible and to further minimize memory and compute wastage on GPUs that are utilized. We have developed two approaches to address this problem: an optimization method and a heuristic method. We benchmark these with two workload scheduling heuristics for multiple use cases. Our results show up to 2.85x improvement in the number of GPUs used and up to 70% reduction in GPU wastage over baseline heuristics. We plan to enable the SRE community to leverage our proposed method in production environments.

Optimal Workload Placement on Multi-Instance GPUs

TL;DR

The paper tackles optimizing Large Language Model (LLM) inference workloads on Multi-Instance GPUs (MIGs) to minimize total GPU usage and wasted compute/memory. It introduces a framework with a placement recommender, migration planner, and executor, and presents two approaches: a global Mixed-Integer Programming (MIP) method and a fast rule-based heuristic, both addressing initial deployment, compaction, and reconfiguration under Dynamic Resource Allocation (DRA). The core contribution is the WPM MIP model and a tractable heuristic, along with preprocessing to handle partially partitioned GPUs; evaluations show up to 2.85x GPU reduction and up to 70% wastage reduction versus baselines, with competitive performance from the heuristic. The work offers a practical path toward production-ready MIG-aware scheduling, enabling non-disruptive migrations and more available GPU resources for future workloads.

Abstract

There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we investigate how to optimize the placement of Large Language Model (LLM)-based AI Inferencing workloads on GPUs. We first identify and present several use cases that are encountered in practice that require workloads to be efficiently placed or migrated to other GPUs to make room for incoming workloads. The overarching goal is to use as few GPUs as possible and to further minimize memory and compute wastage on GPUs that are utilized. We have developed two approaches to address this problem: an optimization method and a heuristic method. We benchmark these with two workload scheduling heuristics for multiple use cases. Our results show up to 2.85x improvement in the number of GPUs used and up to 70% reduction in GPU wastage over baseline heuristics. We plan to enable the SRE community to leverage our proposed method in production environments.
Paper Structure (31 sections, 3 equations, 11 figures, 3 tables, 1 algorithm)

This paper contains 31 sections, 3 equations, 11 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Definitions
  4. Deployments with Multi-Instance GPUs
  5. Nvidia GPU Operation
  6. Dynamic Resource Allocation
  7. Placement Optimization Use Cases
  8. Initial Deployment
  9. Compaction
  10. Reconfiguration/Redeployment
  11. Efficient Placement with MIG under DRA
  12. Objectives of MIG Placement
  13. Objective 1: Minimizing number of GPUs.
  14. Objective 2: Minimizing GPU compute or memory wastage.
  15. Objective 3: Maximizing GPU availability.
  16. ...and 16 more sections

Figures (11)

  • Figure 1: Framework Overview
  • Figure 2: GPU Slices
  • Figure 3: Initial deployment of new workloads
  • Figure 4: Initial deployment or compaction
  • Figure 5: Reconfiguration
  • ...and 6 more figures