Maintaining $k$-MinHash Signatures over Fully-Dynamic Data Streams with Recovery

TL;DR

This work tackles maintaining accurate Jaccard similarity sketches for fully dynamic data streams with recovery by introducing the ell-buffered $k$-MinHash (BMH). It proves memory and time guarantees: each subset requires $O(k \log |U|)$ space, updates cost $O(k \log |U|)$ amortized w.h.p., and a subset's $k$-MinHash can be retrieved in $O(k)$ time with the same quality as recomputing from scratch. The approach supports non-legal streams and uses recovery queries to bound costly rebuilds, achieving competitive performance against the prior global solution BSS20 across synthetic and real datasets. Experimental results show substantial speedups over Vanilla-MinHash, favorable query-time behavior, and improved SE accuracy, highlighting practical applicability for large-scale dynamic graphs and streaming data mining. Overall, the ell-buffered $k$-MinHash provides a practical, provably efficient way to maintain high-quality similarity sketches under arbitrary online updates with recovery.

Abstract

We consider the task of performing Jaccard similarity queries over a large collection of items that are dynamically updated according to a streaming input model. An item here is a subset of a large universe $U$ of elements. A well-studied approach to address this important problem in data mining is to design fast-similarity data sketches. In this paper, we focus on global solutions for this problem, i.e., a single data structure which is able to answer both Similarity Estimation and All-Candidate Pairs queries, while also dynamically managing an arbitrary, online sequence of element insertions and deletions received in input. We introduce and provide an in-depth analysis of a dynamic, buffered version of the well-known $k$-MinHash sketch. This buffered version better manages critical update operations thus significantly reducing the number of times the sketch needs to be rebuilt from scratch using expensive recovery queries. We prove that the buffered $k$-MinHash uses $O(k \log |U|)$ memory words per subset and that its amortized update time per insertion/deletion is $O(k \log |U|)$ with high probability. Moreover, our data structure can return the $k$-MinHash signature of any subset in $O(k)$ time, and this signature is exactly the same signature that would be computed from scratch (and thus the quality of the signature is the same as the one guaranteed by the static $k$-MinHash). Analytical and experimental comparisons with the other, state-of-the-art global solutions for this problem given in [Bury et al.,WSDM'18] show that the buffered $k$-MinHash turns out to be competitive in a wide and relevant range of the online input parameters.

Figures (6)

• Figure 1: A counterexample showing that the naive strategy might produce a configuration of the data structure from which it is not possible to recover the min-hash signature of the set. The naive strategy: a delete operation is managed by removing the element from the buffer, while an insertion is implemented by simply inserting the element in the buffer (if there is enough space). In the example, a sequence of update operations is shown for $k=1$ and $\ell=2$. After such a sequence, a query is performed in order to obtain the MinHash signature of the set but the value returned by the data structure is wrong since it does not coincide the the MinHash of the set.
• Figure 2: (a) shows the average time of computing $n$ inserts followed by $n$ deletions (with $n = 2^{16}$) for different values of $k$, varying the size of the buffer $\ell$ from $5$ to $200$. (b) shows the average number of faults. For readability reason the case $\ell = 1$ is omitted.
• Figure 3: (a) comparison as memory changes of $n = 2^{16}$ insertions followed by $n$ deletions. (b) comparison as memory changes of $n=2^{16}$ queries. The $x$-axes indicates the value $k$ and $c^2$. Notice the log-log scale. (c) time comparison for a sequence of $n$ operations varying the fraction $p$ of query, for fixed memory $k = c^2 = 1024$ and $\ell = \log |U| = 17$.
• Figure 4: RMSE for different values of Jaccard similarity, from $0.1$ to $0.9$. The size of the sketches is $1024 \times \log{\vert U \vert}$ memory words, where $|U| = 2^{17}$. Thus Vanilla-MinHash (MH in the figure) uses $k \times \log{\vert U \vert} = 1024 \times 17$ hash functions. The size of the sets are $\approx 6500$. Each point is the RMSE for $1000$ experiments. The shaded areas represent the standard deviation of the experiments.
• Figure 5: (a), (b) and (c) report the average execution time of $100$ independent experiments consisting in $n$ insertion followed by $n$ deletion, for values of $k = 1, 100, 1000, 2000$ and varying $\ell$ from $2$ to $200$. Similarly, (d), (e) and (f) report the average number of faults for the same experiments.

Theorems & Definitions (12)

• Definition 2.1
• Definition 2.2: approximate $d$-min-wise independent hash functions
• Theorem 2.3: feigenblat-apx-kminwise-soda11
• Definition 4.1: Smallest operator
• Definition 4.2: $\ell$-buffered $k$-MinHash
• Lemma 4.3
• Lemma 4.4
• Lemma 4.5
• Lemma 4.6
• Corollary 4.7