BOA Constrictor: Squeezing Performance out of GPUs in the Cloud via Budget-Optimal Allocation

TL;DR

This work tackles the problem of cost-efficient cloud GPU rental for streaming ML training by formulating it as a budget-constrained scheduling problem. It derives the Budget-Optimal Allocation (BOA) policy, showing that offline optimality in a fixed-width, epoch-based allocation translates into online optimal scheduling under a long-run budget $b$. The BOA Constrictor implementation on AdaptDL decouples heavy optimization from the real-time loop, enabling efficient, near-optimal decisions that significantly reduce average $JCT$ (up to ~2x in large-scale simulations) while preserving budget. The combination of theoretical guarantees, practical system design, and extensive evaluation demonstrates a robust approach to balancing cost and performance in cloud-based ML training, with strong improvements in tail performance and applicability to heterogeneous hardware futures.

Abstract

The past decade has seen a dramatic increase in demand for GPUs to train Machine Learning (ML) models. Because it is prohibitively expensive for most organizations to build and maintain a large GPU cluster, organizations instead choose to rent GPUs from cloud providers. The customer is responsible for devising a policy for (i) deciding how many GPUs to rent at every moment in time to process a stream of ML training jobs and (ii) allocating the rented GPUs among the currently active jobs in the system. Because ML training jobs can be parallelized across different numbers of GPUs, the customer generally has many options for how many GPUs to use for each job. Allocating more GPUs to a single training job will cause the job to complete more quickly. However, the customer pays for each GPU-hour they use, and a training job receives a diminishing marginal benefit from running on additional GPUs. Hence, allocating too many GPUs to a single training job can dramatically increase the overall cost that the customer pays to the cloud provider. This gives rise to a cost-performance tradeoff that customers must balance when running training jobs in the cloud. To balance the cost-performance tradeoff, we develop BOA Constrictor, a new scheduler for ML training jobs which uses a Budget-Optimal Allocation (BOA) policy to squeeze the highest level of performance out of a cloud-deployed GPU cluster given a fixed budget constraint. We explicitly formulate the problem as a budget-constrained scheduling problem and derive the BOA policy which minimizes the average job completion time (JCT) of a stream of arriving jobs subject to the user's budget. For a given budget level, we demonstrate that BOA Constrictor can reduce average JCT by 1.6 times in small-scale implementation experiments and by 2 times in detailed, large-scale simulations compared to state-of-the-art heuristic based schedulers.

Figures (11)

• Figure 1: Our solution, BOA Constrictor balances the cost-performance tradeoff for training in the cloud. While existing policies such as Pollux qiao2021pollux do not explicitly balance the cost-performance tradeoff, BOA Constrictor provides up to a $2\times$ improvement in average JCT and up to a $3\times$ improvement in P95 JCT for a given budget by deriving a budget-optimal allocation policy.
• Figure 2: Performance of a Cifar10 training job on different numbers of GPUs. Changes to job hyperparameters over the course of training cause a job's speedup function to shift. Because the speedup functions are concave and sublinear, training costs increase when the job is parallelized on a larger number of GPUs.
• Figure 3: An illustration of BOA Constrictor and the AdaptDL framework.
• Figure 4: Comparing BOA Constrictor and Pollux with autoscaling running 85 jobs from the workload-1 trace qiao2021pollux. Simulation results match implementation results, showing that BOA Constrictor significantly improves the Pareto frontier of average JCT vs. operating budget.
• Figure 5: GPU usage of BOA Constrictor and Pollux with autoscaling implementations. Both policies use a time-average operating budget of ${\sim}$55 GPUs. BOA Constrictor lowers average JCT by reacting faster and more aggressively to bursts in arrivals.

Theorems & Definitions (22)

• definition 1: Optimal policy, BOA
• lemma 1: No queueing
• lemma 2: Same allocation for same speedup
• proof : Proof Sketch
• definition 2: Fixed-width policy
• lemma 3: Operating budget of a fixed width policy
• proof
• lemma 4
• proof : Proof Sketch
• theorem 1: Optimality