Aleph Filter: To Infinity in Constant Time

TL;DR

Aleph Filter provides a memory vs. FPR trade-offs on par with static filters, and supports all operations (insertions, queries, deletes, etc.) in constant time, no matter how much the data grows.

Abstract

Filter data structures are widely used in various areas of computer science to answer approximate set-membership queries. In many applications, the data grows dynamically, requiring their filters to expand along with the data. However, existing methods for expanding filters cannot maintain stable performance, memory footprint, and false positive rate (FPR) simultaneously. We address this problem with Aleph Filter, which makes the following contributions. (1) It supports all operations (insertions, queries, deletes, etc.) in constant time, no matter how much the data grows. (2) Given an estimate of how much the data will ultimately grow, Aleph Filter provides a memory vs. FPR trade-offs on par with static filters.

Figures (18)

• Figure 1: A Quotient filter stores a fingerprint for each key in a hash table, and it resolves has collisions by organizing fingerprints into runs and clusters. Fingerprints are illustrated in italicized red.
• Figure 2: The Fingerprint Sacrifice method transfers one bit from the fingerprint to the slot address while expanding to evenly distribute the entries across the larger hash table.
• Figure 3: By supporting variable-length fingerprints, InfiniFilter can set long fingerprints to newer entries inserted after expanding.
• Figure 4: InfiniFilter in the Widening Regime gradually increases the slot width across expansions to keep the false positive rate constant.
• Figure 5: InfiniFilter deletes the longest matching fingerprints in the target run to prevent future false negatives.