A Data-Informed Local Subspaces Method for Error-Bounded Lossy Compression of Large-Scale Scientific Datasets

TL;DR

A data-driven scientific data compressor, called Discontinuous Data-informed Local Subspaces (Discontinuous DLS), to improve compression-to-error ratios over data-agnostic compressors and to significantly reduce storage requirements without compromising critical data fidelity.

Abstract

The growing volume of scientific simulation data presents a significant challenge for storage and transfer. Error-bounded lossy compression has emerged as a critical solution for mitigating these challenges, providing a means to reduce data size while ensuring that reconstructed data remains valid for scientific analysis. In this paper, we present a data-driven scientific data compressor, called Discontinuous Data-informed Local Subspaces (Discontinuous DLS), to improve compression-to-error ratios over data-agnostic compressors. This error-bounded compressor leverages localized spatial and temporal subspaces, informed by the underlying data structure, to enhance compression efficiency and preserve key features. The presented technique is flexible and applicable to a wide range of scientific data, including fluid dynamics, environmental simulations, and other high-dimensional, time-dependent datasets. We describe the core principles of the method and demonstrate its ability to significantly reduce storage requirements without compromising critical data fidelity. The technique is implemented in a distributed computing environment using MPI, and its performance is evaluated against state-of-the-art error-bounded compression methods in terms of compression ratio and reconstruction accuracy. This study highlights discontinuous DLS as a promising approach for large-scale scientific data compression in high-performance computing environments, providing a robust solution for managing the growing data demands of modern scientific simulations.

Figures (12)

• Figure 1: The error bound discontinuous-DLS based comporession results is compared to the SZ, MGARD, and continuous-DLS based compression of the 3D turbulent flow.
• Figure 2: Comparison of compression performance (CR versus NRMSE) using three types of basis functions: data-adaptive SVD, fixed cosine, and random bases.
• Figure 3: The training snapshot is used to learn the enrichment functions, while the test snapshot is utilized for parametric study. Although only the $u'$ component is displayed here, all three velocity components ($u'$, $v'$, and $w'$) are included in both training and testing.
• Figure 4: Normalized root mean square errors (NRMSEs) and compression ratios (CRs) for 1024 snapshots of flow past a cylinder, computed using various patch sizes and target error bounds.
• Figure 5: Estimated compression ratios for varying patch sizes at three target error levels: 0.1%, 1%, and 10%. The estimates are based on compression ratios computed from a single snapshot and extrapolated to 1024 snapshots.