Key Compression Limits for $k$-Minimum Value Sketches

TL;DR

It is shown that compression based on ordering the keys and encoding successive differences can offer O(log n) bits per key in expected storage savings, and that the encoding method is near-optimal among all methods to encode a KMVsketch.

Abstract

The $k$-Minimum Values (\kmv) data sketch algorithm stores the $k$ least hash keys generated by hashing the items in a dataset. We show that compression based on ordering the keys and encoding successive differences can offer $O(\log n)$ bits per key in expected storage savings, where $n$ is the number of unique values in the data set. We also show that $O(\log n)$ expected bits saved per key is optimal for any form of compression for the $k$ least of $n$ random values -- that the encoding method is near-optimal among all methods to encode a \kmv sketch. We present a practical method to perform that compression, show that it is computationally efficient, and demonstrate that its average savings in practice is within about five percent of the theoretical minimum based on entropy. We verify that our method outperforms off-the-shelf compression methods, and we demonstrate that it is practical, using real and synthetic data.

Figures (7)

• Figure 1: Illustration of the compression algorithm. In the "Hashes" table, the raw hashes are sorted: the leading one, written in bold $\textbf{1}$, is further to the right for smaller hash values. In Step A, we set $\Delta[0] = h[0]$ and evaluate differences between the sorted hash values. In Step B, we find the largest number of leading zeros in the $\Delta[j]$ which are in general not sorted; this is done by successively taking the $\textsf{OR}$ of each newly computed $\Delta[j] = h[j] - h[j-1]$ and is a number between $0$ and $63$ so can be stored in $6$ bits. In this example, it is in column $6$ and all trailing ones in the bottom row are ignored as we just need the number of leading zeros. Finally, in Step C we set $x = 6$ and chop off the leading bits up to index $6$ (in the lefthand blue dashed-line rectangle), and write only the lower $x$ bits (the solid-line red rectangle).
• Figure 2: Sketch sizes and entropy bound (Theorem \ref{['hkn_thm']}) vs. input cardinality. Sketch enters estimation mode to the right of (\ref{['estimation']}) line which is plotted at the location $n = k$.
• Figure 3: Empirical and numerical compression ratio vs. input cardinality for our compression scheme when in estimation mode.
• Figure 4: Comparing serialization times of an uncompressed sketch and using our compression scheme vs input cardinality. Estimation mode to the right of (\ref{['estimation']}).
• Figure 5: Comparison of all methods on the Kasandr dataset. Chart title is the column name and cardinality. Left column is absolute size and the right hand column is the compression rate. In all cases our method is smaller than competitors.

Theorems & Definitions (8)

• Theorem 1
• Theorem 2
• Theorem 3
• proof
• Theorem 4
• proof
• proof
• proof