Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Design and Operation of Shared Machine Learning Clusters on Campus

Kaiqiang Xu, Decang Sun, Hao Wang, Zhenghang Ren, Xinchen Wan, Xudong Liao, Zilong Wang, Junxue Zhang, Kai Chen

TL;DR

This paper tackles the challenge of efficiently managing shared GPU clusters for academic ML research. It introduces SING, a four-layer, full-stack campus cluster manager that decouples job profiling, adaptation, scheduling, and execution, built atop Slurm with ML-specific optimizations. SING achieves high resource utilization through an optimized FCFS with backfill, pre-initialized environments, and two-dimensional resource granularity, while offering an MLaaS-style user experience and interactive debugging. The authors provide extensive operational data, discuss design choices and limitations, and release open-source resources including code, configurations, and a rich job trace to facilitate deployment of similar campus clusters and guide future work in shared ML infrastructure.

Abstract

Amid the rapid advancements in large machine learning (ML) models, universities worldwide are investing substantial funds and efforts into GPU clusters. However, managing a shared GPU cluster poses a pyramid of challenges, from hardware configuration to resource allocation among users. This paper introduces SING, a full-stack solution designed to streamline the management of shared GPU clusters in academic institutions. Motivated by the pressing need for efficient resource sharing and the challenges posed by limited staffing, we present a comprehensive view of SING's architecture and design choices, which achieves operational efficiency (i.e., low maintenance cost and high resource utilization). We also share experience and insights from the real-world operations of SING, including analysis of its usage patterns and management of incidents and failures. This paper is part of our ongoing effort to improve the management of shared ML clusters. We open-source relevant resources to facilitate the development and operation of similar clusters for ML.

Design and Operation of Shared Machine Learning Clusters on Campus

TL;DR

This paper tackles the challenge of efficiently managing shared GPU clusters for academic ML research. It introduces SING, a four-layer, full-stack campus cluster manager that decouples job profiling, adaptation, scheduling, and execution, built atop Slurm with ML-specific optimizations. SING achieves high resource utilization through an optimized FCFS with backfill, pre-initialized environments, and two-dimensional resource granularity, while offering an MLaaS-style user experience and interactive debugging. The authors provide extensive operational data, discuss design choices and limitations, and release open-source resources including code, configurations, and a rich job trace to facilitate deployment of similar campus clusters and guide future work in shared ML infrastructure.

Abstract

Amid the rapid advancements in large machine learning (ML) models, universities worldwide are investing substantial funds and efforts into GPU clusters. However, managing a shared GPU cluster poses a pyramid of challenges, from hardware configuration to resource allocation among users. This paper introduces SING, a full-stack solution designed to streamline the management of shared GPU clusters in academic institutions. Motivated by the pressing need for efficient resource sharing and the challenges posed by limited staffing, we present a comprehensive view of SING's architecture and design choices, which achieves operational efficiency (i.e., low maintenance cost and high resource utilization). We also share experience and insights from the real-world operations of SING, including analysis of its usage patterns and management of incidents and failures. This paper is part of our ongoing effort to improve the management of shared ML clusters. We open-source relevant resources to facilitate the development and operation of similar clusters for ML.

Paper Structure

This paper contains 27 sections, 11 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivations
  3. Existing Solutions for Shared ML Clusters
  4. Direct Shell Access Sharing
  5. Traditional HPC Solution
  6. Container Platforms
  7. Cluster Schedulers and Computing Frameworks
  8. Motivations and Requirements
  9. Design and Implementation
  10. Layer 1: Job Profile Layer
  11. Layer 2: Adapter Layer
  12. Layer 3: Scheduler Layer
  13. Layer 4: Execution Layer
  14. SING Implementation
  15. Usability and Accessibility
  16. ...and 12 more sections

Figures (11)

  • Figure 1: SING uses 4-layer architecture to enhance the overall efficiency and flexibility in operation.
  • Figure 2: Example job profile where users specify job requirements, code entry point, and configurations.
  • Figure 3: Workflow and functions of the adapter layer.
  • Figure 4: SING's scheduler layer implements mechanisms to improve cluster-wide resource utilization.
  • Figure 5: An end-to-end workflow of job processing in SING, from job submission to execution.
  • ...and 6 more figures