Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MagnifierSketch: Quantile Estimation Centered at One Point

Jiarui Guo, Qiushi Lyu, Yuhan Wu, Haoyu Li, Zhaoqian Yao, Yuqi Dong, Xiaolin Wang, Bin Cui, Tong Yang

TL;DR

This work tackles per-key point-quantile estimation in data streams by introducing MagnifierSketch, a two-stage sketch combining Tower Sketch for infrequent-item filtering with a per-key Value Sketch that uses Value Focus, Distribution Calibration, and Double Filtration. The authors prove unbiasedness under distribution calibration, derive single-key and per-key error bounds, and analyze time/space complexity, demonstrating strong improvements in average error and throughput. Extensive experiments across real and synthetic data, plus RocksDB integration, show MagnifierSketch outperforms state-of-the-art baselines in both single-key and per-key settings while remaining efficient for high-speed streams. The approach offers practical, cache-friendly quantile estimation with real-world applicability in databases and network monitoring.

Abstract

In this paper, we take into consideration quantile estimation in data stream models, where every item in the data stream is a key-value pair. Researchers sometimes aim to estimate per-key quantiles (i.e. quantile estimation for every distinct key), and some popular use cases, such as tail latency measurement, recline on a predefined single quantile (e.g. 0.95- or 0.99- quantile) rather than demanding arbitrary quantile estimation. However, existing algorithms are not specially designed for per-key estimation centered at one point. They cannot achieve high accuracy in our problem setting, and their throughput are not satisfactory to handle high-speed items in data streams. To solve this problem, we propose MagnifierSketch for point-quantile estimation. MagnifierSketch supports both single-key and per-key quantile estimation, and its key techniques are named Value Focus, Distribution Calibration and Double Filtration. We provide strict mathematical derivations to prove the unbiasedness of MagnifierSketch and show its space and time complexity. Our experimental results show that the Average Error (AE) of MagnifierSketch is significantly lower than the state-of-the-art in both single-key and per-key situations. We also implement MagnifierSketch on RocksDB database to reduce quantile query latency in real databases. All related codes of MagnifierSketch are open-sourced and available at GitHub.

MagnifierSketch: Quantile Estimation Centered at One Point

TL;DR

This work tackles per-key point-quantile estimation in data streams by introducing MagnifierSketch, a two-stage sketch combining Tower Sketch for infrequent-item filtering with a per-key Value Sketch that uses Value Focus, Distribution Calibration, and Double Filtration. The authors prove unbiasedness under distribution calibration, derive single-key and per-key error bounds, and analyze time/space complexity, demonstrating strong improvements in average error and throughput. Extensive experiments across real and synthetic data, plus RocksDB integration, show MagnifierSketch outperforms state-of-the-art baselines in both single-key and per-key settings while remaining efficient for high-speed streams. The approach offers practical, cache-friendly quantile estimation with real-world applicability in databases and network monitoring.

Abstract

In this paper, we take into consideration quantile estimation in data stream models, where every item in the data stream is a key-value pair. Researchers sometimes aim to estimate per-key quantiles (i.e. quantile estimation for every distinct key), and some popular use cases, such as tail latency measurement, recline on a predefined single quantile (e.g. 0.95- or 0.99- quantile) rather than demanding arbitrary quantile estimation. However, existing algorithms are not specially designed for per-key estimation centered at one point. They cannot achieve high accuracy in our problem setting, and their throughput are not satisfactory to handle high-speed items in data streams. To solve this problem, we propose MagnifierSketch for point-quantile estimation. MagnifierSketch supports both single-key and per-key quantile estimation, and its key techniques are named Value Focus, Distribution Calibration and Double Filtration. We provide strict mathematical derivations to prove the unbiasedness of MagnifierSketch and show its space and time complexity. Our experimental results show that the Average Error (AE) of MagnifierSketch is significantly lower than the state-of-the-art in both single-key and per-key situations. We also implement MagnifierSketch on RocksDB database to reduce quantile query latency in real databases. All related codes of MagnifierSketch are open-sourced and available at GitHub.

Paper Structure

This paper contains 32 sections, 11 theorems, 16 equations, 18 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Prior Art and Their Limitations
  4. Our Proposed Solution
  5. Key Contributions
  6. Problem Statement And Related Work
  7. Problem Statement
  8. Related Work
  9. Quantile Estimation
  10. Frequency Estimation
  11. MagnifierSketch Algorithm
  12. Single-key Situation when $w=0.5$: Value Focus
  13. Idea of Distribution Calibration
  14. Per-key Situation: Value Sketch
  15. Optimization: Early Screening
  16. ...and 17 more sections

Key Result

Theorem 1

For every value or infinity $v$ inserted into Value Sketch, the distribution of the quantile function at $v$ (w.r.t.$F'$) is uniform distribution on the interval $[0,1]$.

Figures (18)

  • Figure 1: Idea of Distribution Calibration
  • Figure 2: Examples in Single-key Situation
  • Figure 3: Examples in Per-key Situation
  • Figure 4: Effects of $d$
  • Figure 5: Effects of $q$
  • ...and 13 more figures

Theorems & Definitions (26)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • ...and 16 more