ZOR filters: fast and smaller than fuse filters

TL;DR

ZOR filters address the challenge of constructing static probabilistic membership filters that nearly achieve the information-theoretic space lower bound while guaranteeing termination. They introduce a deterministic peeling procedure that abandons a small fraction of keys, forming a main fuse-filter-like structure, and recover standard false-positive-only semantics by storing the abandoned keys in a compact auxiliary fingerprint structure. Empirically, abandonment rates drop below 1% for moderate arities (e.g., $N\ge 5$), yielding overall overhead near $1\%$ above $\log_2(1/\varepsilon)$ with query times around $10^2$ ns; the trade-off is a slower build due to needing explicit incidence data. The approach offers a practical path to near-optimal static filters with fuse-like query performance, albeit at current construction-time cost, making it attractive for immutable storage and large-scale indexing where updates are rare.

Abstract

Probabilistic membership filters support fast approximate membership queries with a controlled false-positive probability $\varepsilon$ and are widely used across storage, analytics, networking, and bioinformatics \cite{chang2008bigtable,dayan2018optimalbloom,broder2004network,harris2020improved,marchet2023scalable,chikhi2025logan,hernandez2025reindeer2}. In the static setting, state-of-the-art designs such as XOR and fuse filters achieve low overhead and very fast queries, but their peeling-based construction succeeds only with high probability, which complicates deterministic builds \cite{graf2020xor,graf2022binary,ulrich2023taxor}. We introduce \emph{ZOR filters}, a deterministic continuation of XOR/fuse filters that guarantees construction termination while preserving the same XOR-based query mechanism. ZOR replaces restart-on-failure with deterministic peeling that abandons a small fraction of keys, and restores false-positive-only semantics by storing the remainder in a compact auxiliary structure. In our experiments, the abandoned fraction drops below $1\%$ for moderate arity (e.g., $N\ge 5$), so the auxiliary handles a negligible fraction of keys. As a result, ZOR filters can achieve overhead within $1\%$ of the information-theoretic lower bound $\log_2(1/\varepsilon)$ while retaining fuse-like query performance; the additional cost is concentrated on negative queries due to the auxiliary check. Our current prototype builds several-fold slower than highly optimized fuse builders because it maintains explicit incidence information during deterministic peeling; closing this optimisation gap is an engineering target.

Figures (5)

• Figure 1: Abandoned fraction $\alpha$ as a function of arity $N$ for several set sizes $n$ (left). Abandoned fraction $\alpha$ as a function of the segment (block) size for several arities $N$ (right).
• Figure 2: Memory efficiency of fuse filters, ZOR filters, and an MPHF+fingerprint baseline (MPHF cost fixed to the information-theoretic minimum $1.44$ bits/key). Fingerprint sizes of $4$, $8$, $16$, $24$, and $32$ bits are tested. For ZOR, the reported false-positive rate accounts for both the main and auxiliary structures.
• Figure 3: Impact of tie-breaking strategies on abandoned fraction (left) and construction time (right) for $n=10$M keys.
• Figure 4: Query time for ZOR filters: comparison of positive and negative queries under different arities $N$ without ordering (left) and with ordering (right).
• Figure 5: Construction time for ZOR filters under varying arity $N$ and set size $n$ (left). Construction time for ZOR filters under varying arity $N$ and partition count $P$ (right).