Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Train Your Filter: Should You Learn, Stack or Adapt?

Diandre Miguel Sabale, Wolfgang Gatterbauer, Prashant Pandey

Abstract

Filters are ubiquitous in computer science, enabling space-efficient approximate membership testing. Since Bloom filters were introduced in 1970, decades of work improved their space efficiency and performance. Recently, three new paradigms have emerged offering orders-of-magnitude improvements in false positive rates (FPRs) by using information beyond the input set: (1) learned filters train a model to distinguish (non)members, (2) stacked filters use negative workload samples to build cascading layers, and (3) adaptive filters update internal representation in response to false positive feedback. Yet each paradigm targets specific use cases, introduces complex configuration tuning, and has been evaluated in isolation. This results in unclear trade-offs and a gap in understanding how these approaches compare and when each is most appropriate. This paper presents the first comprehensive evaluation of learned, stacked, and adaptive filters across real-world datasets and query workloads. Our results reveal critical trade-offs: (1) Learned filters achieve up to 10^2 times lower FPRs but exhibit high variance and lack robustness under skewed or dynamic workloads. Critically, model inference overhead leads to up to 10^4 times slower query latencies than stacked or adaptive filters. (2) Stacked filters reliably achieve up to 10^3 times lower FPRs on skewed workloads but require workload knowledge. (3) Adaptive filters are robust across settings, achieving up to 10^3 times lower FPRs under adversarial queries without workload assumptions. Based on our analysis, learned filters suit stable workloads where input features enable effective model training and space constraints are paramount, stacked filters excel when reliable query distributions are known, and adaptive filters are most generalizable, providing robust theoretically bound guarantees even in dynamic or adversarial environments.

How to Train Your Filter: Should You Learn, Stack or Adapt?

Abstract

Filters are ubiquitous in computer science, enabling space-efficient approximate membership testing. Since Bloom filters were introduced in 1970, decades of work improved their space efficiency and performance. Recently, three new paradigms have emerged offering orders-of-magnitude improvements in false positive rates (FPRs) by using information beyond the input set: (1) learned filters train a model to distinguish (non)members, (2) stacked filters use negative workload samples to build cascading layers, and (3) adaptive filters update internal representation in response to false positive feedback. Yet each paradigm targets specific use cases, introduces complex configuration tuning, and has been evaluated in isolation. This results in unclear trade-offs and a gap in understanding how these approaches compare and when each is most appropriate. This paper presents the first comprehensive evaluation of learned, stacked, and adaptive filters across real-world datasets and query workloads. Our results reveal critical trade-offs: (1) Learned filters achieve up to 10^2 times lower FPRs but exhibit high variance and lack robustness under skewed or dynamic workloads. Critically, model inference overhead leads to up to 10^4 times slower query latencies than stacked or adaptive filters. (2) Stacked filters reliably achieve up to 10^3 times lower FPRs on skewed workloads but require workload knowledge. (3) Adaptive filters are robust across settings, achieving up to 10^3 times lower FPRs under adversarial queries without workload assumptions. Based on our analysis, learned filters suit stable workloads where input features enable effective model training and space constraints are paramount, stacked filters excel when reliable query distributions are known, and adaptive filters are most generalizable, providing robust theoretically bound guarantees even in dynamic or adversarial environments.
Paper Structure (44 sections, 4 theorems, 4 equations, 27 figures, 4 tables)

This paper contains 44 sections, 4 theorems, 4 equations, 27 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Learned filters
  3. Theoretical guarantees
  4. Ada-BF
  5. PLBF
  6. Stacked filters
  7. Adaptive filters
  8. Adaptive quotient filter
  9. Experimental methodology
  10. Research questions
  11. Evaluation framework
  12. Datasets
  13. Filter construction
  14. Training learned filters
  15. Query Generation
  16. ...and 29 more sections

Key Result

Theorem 1

Mitzenmacher2018 Consider a learned Bloom filter with $n$ keys, $b$ bits per key used by the backup filter, and slot fill rate $\alpha$. For its learned function using $\zeta$ bits, let $f_m$ be the false positive probability, and $f_n$ the false negative probability. The learned filter is expected

Figures (27)

  • Figure 1: Before deployment, learned filters train on dataset features and stacked filters train on sampled negative queries. During deployment, adaptive filters update on false positive feedback.
  • Figure 2: Ada-BF structure. Keys are assigned a number of hash functions depending on the model's predicted likelihood of them being a positive element. The bits corresponding to all their hashes are used for insertions and queries.
  • Figure 3: Partitioned learned Bloom filter (PLBF) Structure. Keys are assigned to different backup filters depending on the score assigned to them by a model.
  • Figure 4: Stacked filter structure. Queries undergo membership queries across layers of filters alternately storing positive and negative queries.
  • Figure 5: Adaptive filters update their representation upon detecting a false positive to avoid committing the same false positive again. To adapt they employ an on-disk reverse map to access the original key corresponding to the colliding fingerprint. This reverse map is only accessed upon false positives.
  • ...and 22 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4