Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring

Qian Gong, Jieyang Chen, Ben Whitney, Xin Liang, Viktor Reshniak, Tania Banerjee, Jaemoon Lee, Anand Rangarajan, Lipeng Wan, Nicolas Vidal, Qing Liu, Ana Gainaru, Norbert Podhorszki, Richard Archibald, Sanjay Ranka, Scott Klasky

TL;DR

MGARD tackles the data deluge from exascale simulations and large telescopes by introducing a multigrid adaptive reduction framework for loss-tolerant data compression and data refactoring. It uses a hierarchical representation $u'_{mc}$, quantization, and bitplane encoding, with error estimators guaranteeing fidelity in the $L^2$ norm and for quantities of interest, across structured and unstructured grids. The software features two API tiers (high-level and low-level), auto-tuning, multi-device out-of-core operation, and GPU-accelerated kernels, and it integrates with ADIOS for seamless I/O. Across plasma physics, cosmology, and radio astronomy, MGARD demonstrates dramatic storage reductions and I/O speedups while preserving scientific fidelity, including QoI-preserving post-processing and progressive recomposition via refactoring.

Abstract

We describe MGARD, a software providing MultiGrid Adaptive Reduction for floating-point scientific data on structured and unstructured grids. With exceptional data compression capability and precise error control, MGARD addresses a wide range of requirements, including storage reduction, high-performance I/O, and in-situ data analysis. It features a unified application programming interface (API) that seamlessly operates across diverse computing architectures. MGARD has been optimized with highly-tuned GPU kernels and efficient memory and device management mechanisms, ensuring scalable and rapid operations.

MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring

TL;DR

MGARD tackles the data deluge from exascale simulations and large telescopes by introducing a multigrid adaptive reduction framework for loss-tolerant data compression and data refactoring. It uses a hierarchical representation , quantization, and bitplane encoding, with error estimators guaranteeing fidelity in the norm and for quantities of interest, across structured and unstructured grids. The software features two API tiers (high-level and low-level), auto-tuning, multi-device out-of-core operation, and GPU-accelerated kernels, and it integrates with ADIOS for seamless I/O. Across plasma physics, cosmology, and radio astronomy, MGARD demonstrates dramatic storage reductions and I/O speedups while preserving scientific fidelity, including QoI-preserving post-processing and progressive recomposition via refactoring.

Abstract

We describe MGARD, a software providing MultiGrid Adaptive Reduction for floating-point scientific data on structured and unstructured grids. With exceptional data compression capability and precise error control, MGARD addresses a wide range of requirements, including storage reduction, high-performance I/O, and in-situ data analysis. It features a unified application programming interface (API) that seamlessly operates across diverse computing architectures. MGARD has been optimized with highly-tuned GPU kernels and efficient memory and device management mechanisms, ensuring scalable and rapid operations.
Paper Structure (13 sections, 5 figures, 1 table)

This paper contains 13 sections, 5 figures, 1 table.

Table of Contents

  1. Motivation and significance
  2. Software design
  3. APIs
  4. High-Level APIs
  5. Low-Level APIs
  6. Software architecture
  7. Software functionalities
  8. Illustrative examples
  9. Application impact
  10. Plasma physics
  11. Earth and cosmological science
  12. Radio astronomy
  13. Conclusion

Figures (5)

  • Figure 1: The software pipeline overview illustrating the two primary functionalities of MGARD -- compression and refactoring, both with precision error control.
  • Figure 2: Software architecture of MGARD
  • Figure 3: Illustration of errors in QoIs derived from the XGC f-data lossy compressed by MGARD with QoI post-processing.
  • Figure 4: Comparing the throughput performance of compression and decompression provided by MGARD, cuSZ, and ZFP-CUDA on OLCF Summit nodes, using NYX data and a relative error bound of 1.0e-3.
  • Figure 6: Global distributing of TC tracks detected in hourly, 6-hourly, and spatiotemporally adaptive reduced hourly data over one year time span.