Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds

TL;DR

The paper tackles the data explosion from large-scale scientific simulations by proposing a hierarchical, attention-guided, error-bounded compression framework. It combines a coarse-to-fine pipeline: a self-attention–driven hyper-block autoencoder to capture inter-block correlations, a block-wise residual autoencoder for block-specific detail, and a PCA-based post-processing step to guarantee per-block reconstruction error $||x-x^G||_2 \leq \tau$. Key contributions include the hyper-block attention mechanism, the residual block-wise autoencoder, PCA-based error guarantees with efficient coefficient storage, and entropy coding for latent data; these yield robust, scalable compression across multi-variable and single-variable datasets. The approach demonstrates up to 8× compression on multi-variable S3D and significant gains on E3SM and XGC compared with SZ3 and ZFP, highlighting its practical potential for managing scientific data while preserving fidelity. This work advances reliable data reduction for high-dimensional scientific workflows, enabling faster analysis and reduced storage without compromising critical accuracy bounds.

Abstract

Scientific applications in fields such as high energy physics, computational fluid dynamics, and climate science generate vast amounts of data at high velocities. This exponential growth in data production is surpassing the advancements in computing power, network capabilities, and storage capacities. To address this challenge, data compression or reduction techniques are crucial. These scientific datasets have underlying data structures that consist of structured and block structured multidimensional meshes where each grid point corresponds to a tensor. It is important that data reduction techniques leverage strong spatial and temporal correlations that are ubiquitous in these applications. Additionally, applications such as CFD, process tensors comprising hundred plus species and their attributes at each grid point. Reduction techniques should be able to leverage interrelationships between the elements in each tensor. In this paper, we propose an attention-based hierarchical compression method utilizing a block-wise compression setup. We introduce an attention-based hyper-block autoencoder to capture inter-block correlations, followed by a block-wise encoder to capture block-specific information. A PCA-based post-processing step is employed to guarantee error bounds for each data block. Our method effectively captures both spatiotemporal and inter-variable correlations within and between data blocks. Compared to the state-of-the-art SZ3, our method achieves up to 8 times higher compression ratio on the multi-variable S3D dataset. When evaluated on single-variable setups using the E3SM and XGC datasets, our method still achieves up to 3 times and 2 times higher compression ratio, respectively.

Figures (9)

• Figure 1: Method Overview: The original data is divided into hyper-blocks. An attention-based autoencoder is employed to capture inter-block correlations within these hyper-blocks. The residual data from the hyper-block Autoencoder is then processed block by block using a block-wise autoencoder for finer reconstruction. Finally, the reconstructed data undergoes a PCA-based method to ensure the error of each data patch is within guaranteed bounds.
• Figure 2: Self-Attention Mechanism
• Figure 3: Indices Encoding
• Figure 4: Ablation study of latent size on S3D dataset
• Figure 5: Ablation study of each model component