Error bounded compression for weather and climate applications

TL;DR

Weather and climate simulations generate petabytes of data, necessitating efficient, error-bounded compression. EBCC tackles this with a two-layer scheme: a base JPEG2000 layer for bulk compression and a residual DWT+SPIHT layer guided by a feedback rate-control loop to meet a user-defined bound $\\epsilon$, processing data chunk-by-chunk for scalability. The approach outperforms established lossy compressors across error targets from 0.1% to 10%, achieving $R$-ratios from $15\\times$ to over $300\\times$, and maintaining downstream errors within ERA5 natural variability in energy-budget and Lagrangian-trajectory benchmarks. Its open-source implementation and benchmark-driven validation suggest substantial practical impact for storing and analyzing high-resolution weather and climate data without sacrificing critical downstream quantities.

Abstract

As the resolution of weather and climate simulations increases, the amount of data produced is growing rapidly from hundreds of terabytes to tens of petabytes. The huge size becomes a limiting factor for broader adoption, and its fast growth rate will soon exhaust all the available storage devices. To address these issues, we present EBCC (Error Bounded Climate-data Compressor). It follows a two-layer approach: a base compression layer using JPEG2000 to capture the bulk of the data with a high compression ratio, and a residual compression layer using wavelet transform and SPIHT encoding to efficiently eliminate long-tail extreme errors introduced by the base compression layer. It incorporates a feedback rate-control mechanism for both layers that adjusts compression ratios to achieve the specified maximum error target. We evaluate EBCC alongside other established compression methods on benchmarks related to weather and climate science including error statistics, a case study on primitive and derived variables near a hurricane, evaluation of the closure of the global energy budget, and a Lagrangian air parcel trajectory simulation. This is the first time that trajectory simulation is used to benchmark compression methods. Our method concentrates most errors near zero, while others tend to distribute errors uniformly within the error bound. EBCC outperforms other methods in the benchmarks at relative error targets ranging from 0.1% to 10% and achieves compression ratios from 15x to more than 300x. In the energy budget closure and Lagrangian trajectory benchmarks, it can achieve more than 100x compression while keeping errors within natural variability derived from ERA5 uncertainty members. This verifies the effectiveness of EBCC in creating heavily compressed weather and climate datasets suitable for downstream applications. The source code of EBCC is available in github.com/spcl/EBCC.

Figures (22)

• Figure 1: Diagram of EBCC (Error Bounded Climate Compressor) method. It compresses weather & climate datasets using a two-layer approach (\ref{['sec:method']}): the base compression layer uses JPEG2000 for bulk compression, while the residual compression layer encodes excessive errors in the residue using discrete wavelet transform (DWT) and the set partitioning in hierarchical tree (SPIHT) algorithm. Both layers incorporate feedback control to achieve the target error bound. EBCC is evaluated on a suite of benchmarks and outperforms existing compression methods (\ref{['sec:benchmark']})
• Figure 2: Distribution of compression ratios for different parameter $q$. The compression ratios are presented relative to using pure base compression (JPEG2000) only ($q=1$). Horizontal ticks in the violin plot represent maximum value, 0.999, 0.99, and 0.9 quantiles of compression ratios.
• Figure 3: Left: Line plot of compression ratio (the higher the better) and structural similarity (SSIM, the higher the better) for each compression method (rate-distortion plot). The 0.99995 SSIM threshold proposed by baker2019ssim is marked on the plot. The numbers are grouped according to the compression method and error target (0.1%, 0.5%, 1%, 5%, 10%), and aggregated using harmonic mean for compression ratios and arithmetic mean for structural similarities over all variables, pressure levels, and timesteps. The values for the Bitrounding method are accumulated according to SSIM bins at [0.95, 0.9, 0.99, 0.999, 0.9999, 0.99999, 0.999999]. Right: Compression and decompression throughput for each compression methods. The higher the better. The numbers are aggregated using arithmetic mean over all variables, pressure levels, timesteps and error targets. Error bars represent standard deviations.
• Figure 4: Histogram of compression errors for 2m temperature (t2m) with compression ratios around 10$\times$ (left) and 100$\times$ (right). X-axis is cropped at the maximum compression error of EBCC. SPERR and Bitrounding have excessive errors in the cropped regions. They result in higher maximum errors while reaching a similar compression ratio as other methods.
• Figure 5: Spherical harmonics power spectra of compressed and original data for geopotential (z) at 500hPa with compression ratios around 10$\times$ (left) and 100$\times$ (right).