Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalability Evaluation of HPC Multi-GPU Training for ECG-based LLMs

Dimitar Mileski, Nikola Petrovski, Marjan Gusev

TL;DR

This work evaluates the scalability of training ECG-based transformer language models on HPC resources, comparing intra-node multi-GPU acceleration with inter-node distributed training on MeluXina. Using SLURM, Apptainer, CUDA, and PyTorch, the study trains foundation and fine-tuned models across 16 ECG datasets totaling 272 GB, observing sub-linear speedups: about $1.6\times$ with 2 GPUs and $1.9\times$ with 4 GPUs. The findings highlight the practical feasibility of HPC-driven ECG-Language Model development but also reveal bottlenecks in GPU scaling and data handling that necessitate further optimizations—e.g., DALI integration, MIG considerations, and enhanced multi-node training with PyTorch Distributed. The results inform future directions to more efficiently exploit EuroHPC resources for healthcare AI, potentially accelerating cardiovascular risk assessment and arrhythmia monitoring in clinical settings.

Abstract

Training large language models requires extensive processing, made possible by many high-performance computing resources. This study compares multi-node and multi-GPU environments for training large language models of electrocardiograms. It provides a detailed mapping of current frameworks for distributed deep learning in multinode and multi-GPU settings, including Horovod from Uber, DeepSpeed from Microsoft, and the built-in distributed capabilities of PyTorch and TensorFlow. We compare various multi-GPU setups for different dataset configurations, utilizing multiple HPC nodes independently and focusing on scalability, speedup, efficiency, and overhead. The analysis leverages HPC infrastructure with SLURM, Apptainer (Singularity) containers, CUDA, PyTorch, and shell scripts to support training workflows and automation. We achieved a sub-linear speedup when scaling the number of GPUs, with values of 1.6x for two and 1.9x for four.

Scalability Evaluation of HPC Multi-GPU Training for ECG-based LLMs

TL;DR

This work evaluates the scalability of training ECG-based transformer language models on HPC resources, comparing intra-node multi-GPU acceleration with inter-node distributed training on MeluXina. Using SLURM, Apptainer, CUDA, and PyTorch, the study trains foundation and fine-tuned models across 16 ECG datasets totaling 272 GB, observing sub-linear speedups: about with 2 GPUs and with 4 GPUs. The findings highlight the practical feasibility of HPC-driven ECG-Language Model development but also reveal bottlenecks in GPU scaling and data handling that necessitate further optimizations—e.g., DALI integration, MIG considerations, and enhanced multi-node training with PyTorch Distributed. The results inform future directions to more efficiently exploit EuroHPC resources for healthcare AI, potentially accelerating cardiovascular risk assessment and arrhythmia monitoring in clinical settings.

Abstract

Training large language models requires extensive processing, made possible by many high-performance computing resources. This study compares multi-node and multi-GPU environments for training large language models of electrocardiograms. It provides a detailed mapping of current frameworks for distributed deep learning in multinode and multi-GPU settings, including Horovod from Uber, DeepSpeed from Microsoft, and the built-in distributed capabilities of PyTorch and TensorFlow. We compare various multi-GPU setups for different dataset configurations, utilizing multiple HPC nodes independently and focusing on scalability, speedup, efficiency, and overhead. The analysis leverages HPC infrastructure with SLURM, Apptainer (Singularity) containers, CUDA, PyTorch, and shell scripts to support training workflows and automation. We achieved a sub-linear speedup when scaling the number of GPUs, with values of 1.6x for two and 1.9x for four.

Paper Structure

This paper contains 12 sections, 3 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. State-of-the-art
  3. Multi-GPU: Nvidia
  4. Multi Instance (Containers)-GPU: Nvidia
  5. Multi-Node: Distributed DL
  6. Methods
  7. Workflow architecture
  8. Experiments
  9. Evaluation methodology
  10. Results
  11. Discussion
  12. Conclusion

Figures (6)

  • Figure 1: System Architecture
  • Figure 2: Experiment Workflow
  • Figure 3: Comparison of training times across different CPU and GPU configurations.
  • Figure 4: Comparison of training time, speedup, and ideal speedup across different GPU configurations.
  • Figure 5: Efficiency
  • ...and 1 more figures