NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression

TL;DR

NeurLZ tackles the storage and I/O bottlenecks of large-scale scientific simulations by augmenting conventional lossy compressors with compression-time online learning. It trains per-block lightweight skipping DNNs to memorize and predict residual errors, and incorporates cross-field learning plus an error-regulation mechanism to enforce strict or relaxed bounds. The approach yields up to around $94\%$ bit-rate reduction at equivalent distortion on datasets like Nyx, Miranda, and Hurricane, with substantial improvements in PSNR and perceptual metrics and strong scalability as data dimensions grow. This work demonstrates a scalable, adaptive enhancement to scientific lossy compression that can significantly reduce storage and I/O costs in HPC workflows while preserving data fidelity.

Abstract

Large-scale scientific simulations generate massive datasets, posing challenges for storage and I/O. Traditional lossy compression struggles to advance more in balancing compression ratio, data quality, and adaptability to diverse scientific data features. While deep learning-based solutions have been explored, their common practice of relying on large models and offline training limits adaptability to dynamic data characteristics and computational efficiency. To address these challenges, we propose NeurLZ, a neural method designed to enhance lossy compression by integrating online learning, cross-field learning, and robust error regulation. Key innovations of NeurLZ include: (1) compression-time online neural learning with lightweight skipping DNN models, adapting to residual errors without costly offline pertaining, (2) the error-mitigating capability, recovering fine details from compression errors overlooked by conventional compressors, (3) $1\times$ and $2\times$ error-regulation modes, ensuring strict adherence to $1\times$ user-input error bounds strictly or relaxed 2$\times$ bounds for better overall quality, and (4) cross-field learning leveraging inter-field correlations in scientific data to improve conventional methods. Comprehensive evaluations on representative HPC datasets, e.g., Nyx, Miranda, Hurricane, against state-of-the-art compressors show NeurLZ's effectiveness. During the first five learning epochs, NeurLZ achieves an 89% bit rate reduction, with further optimization yielding up to around 94% reduction at equivalent distortion, significantly outperforming existing methods, demonstrating NeurLZ's superior performance in enhancing scientific lossy compression as a scalable and efficient solution.

Figures (16)

• Figure 1: (Top) compression error anatomy. (Bottom) the file format of NeurLZ compression archive.
• Figure 2: The NeurLZ workflow ( bottom) compared with related work consisting of a conventional compressor and alternative DNN-based processor ( top).
• Figure 3: NeurLZ achieves higher PSNR across all Nyx fields, outperforming both the non-residual and transformer models (left), as well as the non-skipping and single-field learning (SFLZ) models (right).
• Figure 4: Outlier management: decompressed (reconstructed) values are further enhanced with predicted residuals, and outliers are replaced with in-bound decompressed values to reliably satisfy the error bound.
• Figure 5: Regulated DNN output: The red band shows decompressed values, with the worst-case value (yellow star) processed by three DNNs—tight (A), balanced (B), and loose (C) regulation. The blue bands show that Case A misses the original, Case B reaches it accurately, and Case C exceeds the 2$\times$ bound.