Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems

TL;DR

This work addresses the challenge of performing hyperparameter optimization for neural networks on HPC systems with terabyte-scale datasets. It introduces Resource-Adaptive Successive Doubling (RASDA), which combines time-based successive halving with space-based successive doubling, implemented via Ray Tune and PyTorch-DDP to leverage large-scale data-parallel training. Empirical results across CV, CFD, and AM domains show RASDA achieves up to 1.9x runtime speed-ups over ASHA, scales efficiently up to 1,024 GPUs, and often improves final model quality by exploiting increasing batch sizes. The approach enables systematic HPO on massive scientific data, with demonstrated TB-scale applicability and practical compatibility with existing HPO tools, marking a significant advance in scalable, resource-aware HPO on HPC platforms.

Abstract

On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data. The implementation of RASDA is available on https://github.com/olympiquemarcel/rasda

Figures (7)

• Figure 1: Comparison of successive halving in time (top) and halving in time combined with doubling in space (bottom). Each line corresponds to the learning curve of a single HPO combination.
• Figure 2: Comparison of (plain) ASHA, performing successive halving only in the time domain, and RASDA, performing successive halving in the time and successive doubling in the space domain at the same time on an GPU cluster. In the RASDA case, when a trial is terminated, its workers are allocated to the more promising trials to increase the parallelism of the data-parallel training. Black arrows indicate communication of gradients between GPU.
• Figure 3: Comparison of ASHA and RASDA for training a ResNet50 model on ImageNet for 20 epochs on different scales on the JURECA-DC-GPU system.
• Figure 4: Weak scalability of the RASDA algorithm on up to $1{,}024$GPU on the JUWELS BOOSTER system, including ideal scalability for comparison.
• Figure 5: Exemplary comparison of the performance (in terms of validation accuracy, training loss, and validation loss) of the best configuration found by ASHA and RASDA for the different application cases.