QuaRs: A Transform for Better Lossless Compression of Integers

TL;DR

The paper tackles the inefficiency of fast lossless integer compression when data distributions deviate from a zero-centered unimodal shape. It introduces QuaRs, an invertible, quantile-based transform that remaps data so frequent values map to smaller magnitudes, reshaping arbitrary distributions toward unimodality around zero. The method runs in $O(N \log N)$ time for quantile computation and $O(N)$ for applying the transform, with an overall complexity of $O(N \log N + q \log q)$, and a tunable parameter $q$. Empirical results across multimodal, sparse/asymmetric, sinusoidal-with-noise, and Gaussian-noise datasets show improved compression potential while preserving invertibility, enabling seamless integration with existing fast integer coders and benefiting IoT/time-series data workflows.

Abstract

The rise of integer-valued data, partly driven by the Internet of Things (IoT), has increased demand for efficient compression methods to reduce storage and transmission costs. Existing, speed-oriented methods rely on the ``smaller-numbers-less-bits'' principle, assuming unimodal distributions centered around zero. This assumption is often violated in practice, leading to suboptimal compression. We propose QuaRs, a transformation that reshapes arbitrary distributions into unimodal ones centered around zero, improving compatibility with fast integer compression methods. QuaRs remaps data based on quantiles, assigning smaller magnitudes to frequent values. The method is fast, invertible, and has sub-quadratic complexity. QuaRs enhances compression efficiency, even for challenging distributions, while integrating seamlessly with existing techniques.

Figures (1)

• Figure 1: Applying to different numeric data sets.