MalleTrain: Deep Neural Network Training on Unfillable Supercomputer Nodes

TL;DR

MalleTrain addresses the underutilization of idle, unfillable HPC nodes by enabling malleable DNN training at runtime. It introduces a lightweight online Job Profiling Advisor (JPA) to automatically gather scalability data and uses MILP-based resource allocation to reconfigure training tasks on demand, without requiring users to predefine model information. The evaluation on Summit/Polaris traces and real clusters shows significant throughput gains (up to 22.3%) and demonstrates applicability to dynamic workloads such as NAS and HPO. The work provides a practical architecture for harnessing fragmented idle resources for large-scale DNN workloads and suggests broader applicability to infrastructure management tasks beyond HPC.

Abstract

First-come first-serve scheduling can result in substantial (up to 10%) of transiently idle nodes on supercomputers. Recognizing that such unfilled nodes are well-suited for deep neural network (DNN) training, due to the flexible nature of DNN training tasks, Liu et al. proposed that the re-scaling DNN training tasks to fit gaps in schedules be formulated as a mixed-integer linear programming (MILP) problem, and demonstrated via simulation the potential benefits of the approach. Here, we introduce MalleTrain, a system that provides the first practical implementation of this approach and that furthermore generalizes it by allowing it use even for DNN training applications for which model information is unknown before runtime. Key to this latter innovation is the use of a lightweight online job profiling advisor (JPA) to collect critical scalability information for DNN jobs -- information that it then employs to optimize resource allocations dynamically, in real time. We describe the MalleTrain architecture and present the results of a detailed experimental evaluation on a supercomputer GPU cluster and several representative DNN training workloads, including neural architecture search and hyperparameter optimization. Our results not only confirm the practical feasibility of leveraging idle supercomputer nodes for DNN training but improve significantly on prior results, improving training throughput by up to 22.3\% without requiring users to provide job scalability information.

Figures (14)

• Figure 1: Illustration of dynamic fragment resources on a portion of a cluster. At time $t$, there are three idle nodes in the MalleTrain resource pool.
• Figure 2: Example of fragment resources distribution on Polaris (27th in the TOP500 supercomputer list on Nov. 2023). Red stars mark fragmented idle resources scattered on the cluster. Note: For clarity in presentation, the figure depicts a majority of the cluster rather than its entirety.
• Figure 3: Schematic of the MalleTrain architecture. Scavenger adopts idle nodes, Resource Allocator determines a map of nodes to jobs, Job Manager rescales jobs according to the map, Job Monitor tracks job progress, and Job Profiling Advisor manages the online profiling process.
• Figure 4: Event-driven resource allocation process.
• Figure 5: Rescaling overhead costs on Polaris A100 GPU nodes: (a) Time to scale up and down a single node, for different models; (b) Time to scale up different numbers of nodes, for ResNet-50 model.