Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memento Filter: A Fast, Dynamic, and Robust Range Filter

Navid Eslami, Niv Dayan

TL;DR

This work implements Memento filter on top of a Rank-and-Select Quotient filter and InfiniFilter and demonstrates that it achieves a competitive false positive rate and performance with the state of the art while also providing dynamicity.

Abstract

Range filters are probabilistic data structures that answer approximate range emptiness queries. They aid in avoiding processing empty range queries and have use cases in many application domains such as key-value stores and social web analytics. However, current range filter designs do not support dynamically changing and growing datasets. Moreover, several of these designs also exhibit impractically high false positive rates under correlated workloads, which are common in practice. These impediments restrict the applicability of range filters across a wide range of use cases. We introduce Memento filter, the first range filter to offer dynamicity, fast operations, and a robust false positive rate guarantee for any workload. Memento filter partitions the key universe and clusters its keys according to this partitioning. For each cluster, it stores a fingerprint and a list of key suffixes contiguously. The encoding of these lists makes them amenable to existing dynamic filter structures. Due to the well-defined one-to-one mapping from keys to suffixes, Memento filter supports inserts and deletes and can even expand to accommodate a growing dataset. We implement Memento filter on top of a Rank-and-Select Quotient filter and InfiniFilter and demonstrate that it achieves competitive false positive rates and performance with the state-of-the-art while also providing dynamicity. Due to its dynamicity, Memento filter is the first range filter applicable to B-Trees. We showcase this by integrating Memento filter into WiredTiger, a B-Tree-based key-value store. Memento filter doubles WiredTiger's range query throughput when 50% of the queries are empty while keeping all other cost metrics unharmed.

Memento Filter: A Fast, Dynamic, and Robust Range Filter

TL;DR

This work implements Memento filter on top of a Rank-and-Select Quotient filter and InfiniFilter and demonstrates that it achieves a competitive false positive rate and performance with the state of the art while also providing dynamicity.

Abstract

Range filters are probabilistic data structures that answer approximate range emptiness queries. They aid in avoiding processing empty range queries and have use cases in many application domains such as key-value stores and social web analytics. However, current range filter designs do not support dynamically changing and growing datasets. Moreover, several of these designs also exhibit impractically high false positive rates under correlated workloads, which are common in practice. These impediments restrict the applicability of range filters across a wide range of use cases. We introduce Memento filter, the first range filter to offer dynamicity, fast operations, and a robust false positive rate guarantee for any workload. Memento filter partitions the key universe and clusters its keys according to this partitioning. For each cluster, it stores a fingerprint and a list of key suffixes contiguously. The encoding of these lists makes them amenable to existing dynamic filter structures. Due to the well-defined one-to-one mapping from keys to suffixes, Memento filter supports inserts and deletes and can even expand to accommodate a growing dataset. We implement Memento filter on top of a Rank-and-Select Quotient filter and InfiniFilter and demonstrate that it achieves competitive false positive rates and performance with the state-of-the-art while also providing dynamicity. Due to its dynamicity, Memento filter is the first range filter applicable to B-Trees. We showcase this by integrating Memento filter into WiredTiger, a B-Tree-based key-value store. Memento filter doubles WiredTiger's range query throughput when 50% of the queries are empty while keeping all other cost metrics unharmed.
Paper Structure (11 sections, 1 theorem, 14 figures, 2 tables)

This paper contains 11 sections, 1 theorem, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Analysis
  3. Background
  4. Memento Filter
  5. Expandability
  6. Theoretical Analysis
  7. Evaluation
  8. Standalone Evaluation
  9. B-Tree Evaluation
  10. Conclusion
  11. Expected Cluster Size

Key Result

Theorem 6.1

$\mathbb{E}[|\mathcal{C}|] \leq \frac{\alpha \gamma(\ell)}{(1 - e^{-\alpha / \ell}) \cdot (1 - \alpha) \cdot (\gamma(\ell) - \alpha)}$, where $\gamma(\ell) = \frac{\ell}{\beta(\ell)}$.

Figures (14)

  • Figure 1: An RSQF handles hash collisions by pushing fingerprints to the right using Robin Hood Hashing.
  • Figure 2: Each key $k$ is split into a prefix $p(k)$ and a memento $m(k)$. Prefixes partition the key universe and cluster the keys, while mementos denote the positions of keys in their partitions. Each circle in (B) represents a key in the key set.
  • Figure 3: Clusters are comprised of runs, where all but the first are shifted to the right. A run contains one or more keepsake boxes. The single cluster in the illustration has three runs, and the runs contain three, two, and one keepsake box, respectively.
  • Figure 4: Each keepsake box is encoded using one of three cases, depending on how many mementos it contains.
  • Figure 5: A range query $q=[q_l, q_r]$ spans (A) one or (B) two partitions. The keys of a partition are represented as circles, and the checked lower bound, largest, and smallest keys are highlighted in bold.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Theorem 6.1