Lossless compression of simulated radio interferometric visibilities
A. R. Offringa, R. J. van Weeren
TL;DR
This paper introduces Simulated Signal Compression (Sisco), a lossless compression framework for forward-predicted, noiseless radio interferometric visibilities. Sisco combines two-dimensional polynomial-based prediction, mantissa-exponent residual decomposition, strategic data grouping, and deflate encoding to exploit smoothness in time and frequency, achieving substantial size reductions while remaining fully reversible. Across diverse LOFAR, MeerKAT, and MWA datasets, linear, quadratic, or their combinations yield the best results, with average compression around 24% of the original data and up to over 13x total reduction when paired with baseline-dependent averaging (BDA). Integrated as an open-source Casacore storage manager, Sisco enables real-time-like, lossless compression within existing pipelines, offering a practical path to managing the growing data volumes in modern and upcoming radio interferometry, including the SKA. The authors also discuss a potential lossy extension and future performance enhancements, such as GPU acceleration, to further improve throughput while preserving data integrity in calibration workflows.
Abstract
Context. Processing radio interferometric data often requires storing forward-predicted model data. In direction-dependent calibration, these data may have a volume an order of magnitude larger than the original data. Existing lossy compression techniques work well for observed, noisy data, but cause issues in calibration when applied to forward-predicted model data. Aims. To reduce the volume of forward-predicted model data, we present a lossless compression method called Simulated Signal Compression (Sisco) for noiseless data that integrates seamlessly with existing workflows. We show that Sisco can be combined with baseline-dependent averaging for further size reduction. Methods. Sisco decomposes complex floating-point visibility values and uses polynomial extrapolation in time and frequency to predict values, groups bytes for efficient encoding, and compresses residuals using the Deflate algorithm. We evaluate Sisco on diverse LOFAR, MeerKAT, and MWA datasets with various extrapolation functions. Implemented as an open-source Casacore storage manager, it can directly be used by any observatory that makes use of this format. Results. We find that a combination of linear and quadratic prediction yields optimal compression, reducing noiseless forward-predicted model data to 24% of its original volume on average. Compression varies by dataset, ranging from 13% for smooth data to 38% for less predictable data. For pure noise data, compression achieves just a size of 84% due to the unpredictability of such data. With the current implementation, the achieved compression throughput is with 534 MB/s mostly dominated by I/O on our testing platform, but occupies the processor during compression or decompression. Finally, we discuss the extension to a lossy algorithm.