Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exascale In-situ visualization for Astronomy & Cosmology

Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Emiliano Tramontana

TL;DR

The paper tackles the I/O bottlenecks of petabyte-scale Astronomy & Cosmology data by developing an in-situ visualization workflow that streams ChaNGa simulation output through Hecuba to visualization tools. It implements a lambda-architecture streaming layer, adds in-memory data paths in VisIVO (VSTableMem) and a ParaView plugin to enable real-time, on-the-fly analysis without intermediate disk writes. Preliminary results show meaningful speedups over file-based workflows, with VisIVO achieving up to ~39% and ~24% gains, and ParaView showing ~70–79% gains on large datasets, validating the approach. The work demonstrates a viable path toward scalable, real-time visualization at exascale, with planned future enhancements to extend in-memory capabilities and broaden evaluation."

Abstract

Modern simulations and observations in Astronomy & Cosmology (A&C) produce massively large data volumes, posing significant challenges for storage, access and data analysis. A long-standing bottleneck in high-performance computing, especially now in the exascale era, has been the requirement to write these large datasets to disks, which limits the performance. A promising solution to this challenge is in-situ processing, where analysis and visualization are performed concurrently with the simulation itself, bypassing the storage of the simulation data. In this work, we present new results from an approach for in-situ processing based on Hecuba, a framework that provides a highly distributed database for streaming A&C simulation data directly into the visualization pipeline to make possible on-line visualization. By integrating Hecuba with the high-performance cosmological simulator ChaNGa, we enable real-time, in-situ visualization of N-body simulation results using tools such as ParaView and VisIVO.

Exascale In-situ visualization for Astronomy & Cosmology

TL;DR

The paper tackles the I/O bottlenecks of petabyte-scale Astronomy & Cosmology data by developing an in-situ visualization workflow that streams ChaNGa simulation output through Hecuba to visualization tools. It implements a lambda-architecture streaming layer, adds in-memory data paths in VisIVO (VSTableMem) and a ParaView plugin to enable real-time, on-the-fly analysis without intermediate disk writes. Preliminary results show meaningful speedups over file-based workflows, with VisIVO achieving up to ~39% and ~24% gains, and ParaView showing ~70–79% gains on large datasets, validating the approach. The work demonstrates a viable path toward scalable, real-time visualization at exascale, with planned future enhancements to extend in-memory capabilities and broaden evaluation."

Abstract

Modern simulations and observations in Astronomy & Cosmology (A&C) produce massively large data volumes, posing significant challenges for storage, access and data analysis. A long-standing bottleneck in high-performance computing, especially now in the exascale era, has been the requirement to write these large datasets to disks, which limits the performance. A promising solution to this challenge is in-situ processing, where analysis and visualization are performed concurrently with the simulation itself, bypassing the storage of the simulation data. In this work, we present new results from an approach for in-situ processing based on Hecuba, a framework that provides a highly distributed database for streaming A&C simulation data directly into the visualization pipeline to make possible on-line visualization. By integrating Hecuba with the high-performance cosmological simulator ChaNGa, we enable real-time, in-situ visualization of N-body simulation results using tools such as ParaView and VisIVO.

Paper Structure

This paper contains 11 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Data, Tools and Methodology
  4. ChaNGa Cosmological Data
  5. Hecuba
  6. ParaView
  7. VisIVO
  8. Preliminary Results
  9. VisIVO Results
  10. ParaView plugin Results
  11. Conclusions and future work

Figures (3)

  • Figure 1: Hecuba architecture implementing an Object Mapper for Apache Cassandra support both off-line and on-line ChaNGa data analysis.
  • Figure 2: Typical visualization pipeline of VisIVO Server consisting on the application of the three main modules.
  • Figure 3: ParaView application window with the plugin with Hecuba integrated that supports in situ analysis.