Fingerprint Filters Are Optimal

TL;DR

This work establishes a tight information-theoretic lower bound for dynamic filters, proving that any such data structure with universe $U$, capacity $n$, and false-positive rate $\varepsilon = o(1)$ must use at least $n \log \varepsilon^{-1} + n \log e - o(n)$ bits of space, independent of operation time. The authors develop a multi-layer proof: a warmup under history-independence and monotonicity using a two-filter (F and G) communication protocol, then an extension to history-dependent and non-monotone settings via obfuscation sequences and a reconstructible-set framework. The core technical contribution is showing that, no matter how an implementation evolves under dynamic updates, the information-theoretic cost cannot beat the $n \log \varepsilon^{-1} + n \log e$ overhead. The results imply that practical dynamic fingerprints carry a fundamental space overhead of about $1.44n$ bits beyond the vanilla $n \log \varepsilon^{-1}$ term, confirming the optimality of fingerprint-based dynamic filters in a broad sense and guiding future research on tight bounds for special regimes and related retrieval problems.

Abstract

Dynamic filters are data structures supporting approximate membership queries to a dynamic set $S$ of $n$ keys, allowing a small false-positive error rate $\varepsilon$, under insertions and deletions to the set $S$. Essentially all known constructions for dynamic filters use a technique known as fingerprinting. This technique, which was first introduced by Carter et al. in 1978, inherently requires $$\log \binom{n \varepsilon^{-1}}{n} = n \log \varepsilon^{-1} + n \log e - o(n)$$ bits of space when $\varepsilon = o(1)$. Whether or not this bound is optimal for all dynamic filters (rather than just for fingerprint filters) has remained for decades as one of the central open questions in the area. We resolve this question by proving a sharp lower bound of $n \log \varepsilon^{-1} + n \log e - o(n)$ bits for $\varepsilon = o(1)$, regardless of operation time.

Figures (2)

• Figure 1: Obfuscating tree of depth $b = 3$. Each node has a randomly chosen number of children from the range $[1,3]$. The rightmost path, represented as $(v_0, v_1, v_2, v_3)$, has edge labels $X_1, X_2, X_3$.
• Figure 2: Operational sequences $\sigma_{G_1}$ and $\sigma_{F_1}$. The sequence $\sigma_{G_1}$ follows a DFS traversal of $T$ and stops before traversing $(v_1, v_2)$, while the sequence $\sigma_{F_1}$ terminates just after traversing $(v_2, v_1)$.

Theorems & Definitions (30)

• Theorem 1.1
• Definition 2.1: Dynamic filters
• Theorem 3.1
• Proposition 3.1
• Claim 3.2
• proof
• Claim 3.3
• proof
• Lemma 3.4
• proof : Proof of \ref{['lem:choice_of_s']}