Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hidden Sketch: A Space-Efficient Reversible Sketch for Tracking Frequent Items in Data Streams

Zicang Xu, Yuxuan Tian, Yuhan Wu, Tong Yang

TL;DR

This paper introduces Hidden Sketch, a space-efficient reversible data structure for key and frequency encoding that uniquely combines a Reversible Bloom Filter and a Count-Min Sketch for invertible key and frequency storage, enabling precise reconstruction for both keys and their frequencies with minimal memory.

Abstract

Modern data stream applications demand memory-efficient solutions for accurately tracking frequent items, such as heavy hitters and heavy changers, under strict resource constraints. Traditional sketches face inherent accuracy-memory trade-offs: they either lose precision to reduce memory usage or inflate memory costs to enable high recording capacity. This paper introduces Hidden Sketch, a space-efficient reversible data structure for key and frequency encoding. Our design uniquely combines a Reversible Bloom Filter (RBF) and a Count-Min (CM) Sketch for invertible key and frequency storage, enabling precise reconstruction for both keys and their frequencies with minimal memory. Theoretical analysis establishes Hidden Sketch's space complexity and guaranteed reversibility, while extensive experiments demonstrate its substantial improvements in accuracy and space efficiency in frequent item tracking tasks. By eliminating the trade-off between reversibility and space efficiency, Hidden Sketch provides a scalable foundation for real-time stream analytics in resource-constrained environments.

Hidden Sketch: A Space-Efficient Reversible Sketch for Tracking Frequent Items in Data Streams

TL;DR

This paper introduces Hidden Sketch, a space-efficient reversible data structure for key and frequency encoding that uniquely combines a Reversible Bloom Filter and a Count-Min Sketch for invertible key and frequency storage, enabling precise reconstruction for both keys and their frequencies with minimal memory.

Abstract

Modern data stream applications demand memory-efficient solutions for accurately tracking frequent items, such as heavy hitters and heavy changers, under strict resource constraints. Traditional sketches face inherent accuracy-memory trade-offs: they either lose precision to reduce memory usage or inflate memory costs to enable high recording capacity. This paper introduces Hidden Sketch, a space-efficient reversible data structure for key and frequency encoding. Our design uniquely combines a Reversible Bloom Filter (RBF) and a Count-Min (CM) Sketch for invertible key and frequency storage, enabling precise reconstruction for both keys and their frequencies with minimal memory. Theoretical analysis establishes Hidden Sketch's space complexity and guaranteed reversibility, while extensive experiments demonstrate its substantial improvements in accuracy and space efficiency in frequent item tracking tasks. By eliminating the trade-off between reversibility and space efficiency, Hidden Sketch provides a scalable foundation for real-time stream analytics in resource-constrained environments.
Paper Structure (29 sections, 4 theorems, 4 equations, 6 figures, 2 algorithms)

This paper contains 29 sections, 4 theorems, 4 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Our Solution
  4. Background and Related Work
  5. Problem Statement
  6. Related Work
  7. Explicit key recording:
  8. Implicit key recording:
  9. Summary
  10. Hidden Sketch Design
  11. Reversible Bloom Filter(RBF)
  12. Data Structure:
  13. Key Set Recovery:
  14. Frequency Decoding with CM Sketch
  15. Optimization
  16. ...and 14 more sections

Key Result

lemma 1

To achieve a false positive rate $\epsilon$ when storing $n$ keys in a Bloom Filter, the minimize required bit size $m$ of the Bloom Filter is $m=-\frac{n\ln (\epsilon)}{\ln(2)^2}\approx -1.44n\log_2(\epsilon)$.

Figures (6)

  • Figure 1: An example of Reversible Bloom Filter with 32-bit key.
  • Figure 2: Two-stage framework for tracking frequent items.
  • Figure 3: Accuracy comparison of our algorithm with baselines on CAIDA.
  • Figure 4: Accuracy comparison of our algorithm with baselines on MAWI.
  • Figure 5: Accuracy comparison of our algorithm with baselines on IMC.
  • ...and 1 more figures

Theorems & Definitions (4)

  • lemma 1
  • theorem 1
  • theorem 2
  • theorem 3