Performance comparison of Dask and Apache Spark on HPC systems for Neuroimaging

TL;DR

This study evaluates whether Apache Spark or Dask is better suited for neuroimaging workloads on HPC systems by benchmarking four applications (three synthetic and one BIDS App) on a 606 GiB BigBrain dataset and additional CoRR data. Using a Slurm-based HPC cluster with Lustre, the authors vary nodes, file counts, and iterations to compare in-memory, data-local, and lazy-evaluation behaviors, finding no substantial overall difference in performance between the engines. Spark generally consumes more memory and can suffer from memory-related restarts, while Dask often shows higher scheduling overhead; however, the total execution time is largely balanced by data transfer times and I/O contention. The results underscore that data transfer time and memory footprint dominate performance and suggest adopting Dask for Python-centric neuroimaging workflows, while highlighting the need to reduce IO bottlenecks and optimize memory management in HPC deployments. These insights provide practical guidance for selecting Big Data engines in neuroimaging pipelines and point to directions for improving scalability on HPC systems.

Abstract

The general increase in data size and data sharing motivates the adoption of Big Data strategies in several scientific disciplines. However, while several options are available, no particular guidelines exist for selecting a Big Data engine. In this paper, we compare the runtime performance of two popular Big Data engines with Python APIs, Apache Spark, and Dask, in processing neuroimaging pipelines. Our experiments use three synthetic \HL{neuroimaging} applications to process the \SI{606}{\gibi\byte} BigBrain image and an actual pipeline to process data from thousands of anatomical images. We benchmark these applications on a dedicated HPC cluster running the Lustre file system while using varying combinations of the number of nodes, file size, and task duration. Our results show that although there are slight differences between Dask and Spark, the performance of the engines is comparable for data-intensive applications. However, Spark requires more memory than Dask, which can lead to slower runtime depending on configuration and infrastructure. In general, the limiting factor was the data transfer time. While both engines are suitable for neuroimaging, more efforts need to be put to reduce the data transfer time and the memory footprint of applications.

Figures (10)

• Figure 1: Task graph for Incrementation with 3 iterations and 3 BigBrain blocks.
• Figure 2: Task graph for Histogram with 3 BigBrain blocks.
• Figure 3: Increment: varying nodes -- 5000 files, 5 iterations
• Figure 4: Increment: varying iterations -- 4 nodes, 5000 files
• Figure 5: Increment: varying # of files -- 4 nodes, 5 iterations