GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusters

TL;DR

GOGH tackles energy-efficiently scheduling deep learning workloads on heterogeneous GPU clusters where hardware generations vary and upgrades are impractical. It introduces a correlation-guided orchestration framework that uses two neural networks (P1 for initial throughput estimation and P2 for refinement) and an ILP-based optimizer to assign job combinations to GPUs, guided by historical performance data stored in a Catalog. The approach leverages inter-job and inter-GPU correlations, updating predictions with runtime measurements to improve future allocations, including unseen hardware and co-location scenarios. Experiments on a public heterogeneous GPU dataset show improved scheduling efficiency and throughput prediction accuracy, with end-to-end MAE reductions and estimates as low as 5% error, highlighting practical impact for energy-aware DL deployments.

Abstract

The growing demand for computational resources in machine learning has made efficient resource allocation a critical challenge, especially in heterogeneous hardware clusters where devices vary in capability, age, and energy efficiency. Upgrading to the latest hardware is often infeasible, making sustainable use of existing, mixed-generation resources essential. In this paper, we propose a learning-based architecture for managing machine learning workloads in heterogeneous clusters. The system operates online, allocating resources to incoming training or inference requests while minimizing energy consumption and meeting performance requirements. It uses two neural networks: the first provides initial estimates of how well a new model will utilize different hardware types and how it will affect co-located models. An optimizer then allocates resources based on these estimates. After deployment, the system monitors real performance and uses this data to refine its predictions via a second neural network. This updated model improves estimates not only for the current hardware but also for hardware not initially allocated and for co-location scenarios not yet observed. The result is an adaptive, iterative approach that learns over time to make more effective resource allocation decisions in heterogeneous deep learning clusters.

Figures (3)

• Figure 1: The architecture of GOGH.
• Figure 2: Mean Absolute Error (MAE) of different models at initial estimation and estimation refinement.
• Figure 3: Combined validation results of P1-P2 model pairs.