Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In Situ In Transit Hybrid Analysis with Catalyst-ADIOS2

François Mazen, Louis Gombert, Lucas Givord, Charles Gueunet

TL;DR

An innovative approach to limit the required bandwidth when transferring data during in transit analysis is presented, which leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code.

Abstract

In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community.

In Situ In Transit Hybrid Analysis with Catalyst-ADIOS2

TL;DR

An innovative approach to limit the required bandwidth when transferring data during in transit analysis is presented, which leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code.

Abstract

In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community.
Paper Structure (5 sections, 4 figures, 1 table)

This paper contains 5 sections, 4 figures, 1 table.

Table of Contents

  1. Context
  2. In Situ and In Transit Hybrid Analysis
  3. Use Case and Benchmark
  4. Limitations and Perspectives
  5. Conclusion

Figures (4)

  • Figure 1: Comparison of data flow between in transit solution with Catalyst-ADIOS2 (a) and our new hybrid in situ in transit approach (b), where only reduced data are sent over the network.
  • Figure 2: Schematic timeline comparison of in transit analysis (a) and our new hybrid in situ in transit approach (b). Performance gain is expected when data reduction time and reduced data transfer time (b) is less than full data transfer time (a).
  • Figure 3: Results of the 'slice' (left) and 'resampled volume' (right) hybrid rendering pipelines for timestep 1662 for a reduced $50^3$ cell simulation mesh
  • Figure 4: Comparison of the time taken by data reduction and data transfer for both pipelines on the simulation cluster in average by timestep, for the hybrid and the in transit cases.