Learned Data Compression: Challenges and Opportunities for the Future

TL;DR

This work investigates learned data compression for lossless, sorted integer keys, framing the approach as learning an ICDF mapping with an error-bounded piecewise-linear model and residuals to guarantee exact reconstruction. It presents the epsilon-PLA methodology, analyzes its relationship to learned indexing, and reports SIMD-optimized benchmarks showing competitive decompression throughput against state-of-the-art compressors. The authors outline concrete application scenarios across inverted indexes, KV stores, DBMS queries, and vector databases, while detailing critical challenges—hyper-parameter tuning, dynamic updates, model choice, floating-point extension, and hardware acceleration. Overall, the paper argues that learned compression can become a high-performance, flexible foundation for modern data systems, pending further optimization and integration work.

Abstract

Compressing integer keys is a fundamental operation among multiple communities, such as database management (DB), information retrieval (IR), and high-performance computing (HPC). Recent advances in \emph{learned indexes} have inspired the development of \emph{learned compressors}, which leverage simple yet compact machine learning (ML) models to compress large-scale sorted keys. The core idea behind learned compressors is to \emph{losslessly} encode sorted keys by approximating them with \emph{error-bounded} ML models (e.g., piecewise linear functions) and using a \emph{residual array} to guarantee accurate key reconstruction. While the concept of learned compressors remains in its early stages of exploration, our benchmark results demonstrate that an SIMD-optimized learned compressor can significantly outperform state-of-the-art CPU-based compressors. Drawing on our preliminary experiments, this vision paper explores the potential of learned data compression to enhance critical areas in DBMS and related domains. Furthermore, we outline the key technical challenges that existing systems must address when integrating this emerging methodology.

Figures (7)

• Figure 1: (left) Illustration of the disproportionate growth between the device memory capacities and their associated costs. (right) Key design objectives for a compression algorithm: ❶ high compression ratio, ❷ seamless integration with existing systems, and ❸ efficient query processing, ideally enabling direct query processing on the compressed data.
• Figure 2: A toy example of learned integer compressor based on an error-bounded PLA model. ❶ Encoding Stage: By setting $\epsilon=4$, a PLA model with 3 line segments can be fitted such that each residual $|\delta_i|=|\mathcal{K}[i]-\lfloor f(i) \rfloor|\leq\epsilon$. Segments $f_1,f_2,f_3$ and all residuals, $\Delta=\{1, 0, \cdots, 3\}$, are materialized as a compressed version to the original key set $\{1, 3, \cdots, 77\}$. Encoding each residual requires $\lceil\log_2(2\epsilon+1)\rceil=4$ bits. ❷ Decoding Stage: For each index $i\in[0, 5]$, the original key can be losslessly recovered by $\mathcal{K}[i]=\lfloor f_1(i)\rfloor+\Delta[i]$. Similarly, $\mathcal{K}[i]=\lfloor f_2(i)\rfloor+\Delta[i]$ for $i\in[6, 9]$, and $\mathcal{K}[i]=\lfloor f_3(i)\rfloor+\Delta[i]$ for $i\in[10, 20]$.
• Figure 3: Preliminary benchmark results for the original learned compressor implementation boffa2022learned (lc, a.k.a. la-vector) and our SIMD-based optimization (lc-simd). Note we enable the compiler's auto-vectorization for lc. All the other baselines are chosen from a recent benchmark for inverted index compression yan2009inverted. All the experiments are performed on a Ubuntu machine with an Intel© Xeon™ Gold 6430 CPU and 512 GiB DDR5 memory.
• Figure 4: (a) Skip pointers in conventional inverted index designs. (b) Natural pruning support in learned compressors. The possible key range covered by a line segment is known due to the error-bounded nature of an $\epsilon$-PLA.
• Figure 5: Illustration of a vector quantizer (VQ). When product quantization (PQ) is applied, dense vectors are partitioned into subspaces and clustered within each subspace. Each integer in the learned codebook corresponds to a cluster ID.

Theorems & Definitions (2)

• Definition 1: Lossless Integer Compression
• Definition 2: $\epsilon$-PLA