Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Scalable In Transit Solution for Comprehensive Exploration of Simulation Data

Paascal Grosset, James Ahrens

TL;DR

The paper addresses the challenge of storing massive HPC simulation data and enabling exploratory visualization without upfront resource planning. It introduces SeerX, a lightweight, elastic in-transit in situ service built on the Mochi framework that decouples MPI and supports on-the-fly data reduction via BLOSC and SZ3. Key contributions include dynamic elastic resource scaling across allocations, seamless data access through Mochi Yokan and a SeerX MPI Helper, and demonstrable compression throughput and fidelity. This approach enables scalable post hoc exploration of large simulations with reduced I/O and flexible integration with existing visualization pipelines.

Abstract

As simulations produce more data than available disk space on supercomputers, many simulations are employing in situ analysis and visualization to reduce the amount of data that needs to be stored. While in situ visualization offers potential for substantial data reduction, its efficacy is hindered by the need for a priori knowledge. First, we need to know what visualization parameters to use to highlight features of interest. Second, we do not know ahead of time how much resources will be needed to run the in situ workflows, e.g. how many compute nodes will be needed for in situ work. In this work, we present SeerX, a lightweight, scalable in-transit in situ service that supports dynamic resource allocation and lossy compression of 3D simulation data. SeerX enables multiple simulations to offload analysis to a shared, elastic service infrastructure without MPI synchronization.

A Scalable In Transit Solution for Comprehensive Exploration of Simulation Data

TL;DR

The paper addresses the challenge of storing massive HPC simulation data and enabling exploratory visualization without upfront resource planning. It introduces SeerX, a lightweight, elastic in-transit in situ service built on the Mochi framework that decouples MPI and supports on-the-fly data reduction via BLOSC and SZ3. Key contributions include dynamic elastic resource scaling across allocations, seamless data access through Mochi Yokan and a SeerX MPI Helper, and demonstrable compression throughput and fidelity. This approach enables scalable post hoc exploration of large simulations with reduced I/O and flexible integration with existing visualization pipelines.

Abstract

As simulations produce more data than available disk space on supercomputers, many simulations are employing in situ analysis and visualization to reduce the amount of data that needs to be stored. While in situ visualization offers potential for substantial data reduction, its efficacy is hindered by the need for a priori knowledge. First, we need to know what visualization parameters to use to highlight features of interest. Second, we do not know ahead of time how much resources will be needed to run the in situ workflows, e.g. how many compute nodes will be needed for in situ work. In this work, we present SeerX, a lightweight, scalable in-transit in situ service that supports dynamic resource allocation and lossy compression of 3D simulation data. SeerX enables multiple simulations to offload analysis to a shared, elastic service infrastructure without MPI synchronization.
Paper Structure (12 sections, 7 figures)

This paper contains 12 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. In Situ Frameworks
  4. Data Reduction
  5. Architecture
  6. Elastic Resource Scaling
  7. Integrated Data Reduction
  8. Data Access
  9. The SeerX Workflow
  10. Results and Discussion
  11. Conclusion
  12. Acknowledgment

Figures (7)

  • Figure 1: Halo (cluster of particles) in the HACC cosmology simulation viewed from different angles using different color maps. In (b) we see that the large halo has many small halos attached to it.
  • Figure 2: Overview of the SeerX architecture with two simulations are connected and sending data to the service. The SeerX service can span multiple allocations (two in this case) and in situ applications can connect to multiple nodes with databases as needed. Users can also connect to SeerX in real time as well, through the Trame to interact with the data.
  • Figure 3: Compression ratios achieved with SZ3 for particle coordinates ($x$, $y$, $z$) using an absolute error bound of 0.003.
  • Figure 4: Measured throughput for compressing HACC data was about 16 MB/s.
  • Figure 5: Scaling test results for data insertion "Put", and retrieval "Get" performance in Yokan databases. Both operations scale effectively as data sizes increase.
  • ...and 2 more figures