Sublinear Sketches for Approximate Nearest Neighbor and Kernel Density Estimation

TL;DR

The paper develops sublinear sketches for two core streaming problems: ANN and A-KDE, enabling efficient similarity search and density estimation on massive, evolving data. It shows that under a Poisson-distributed input model, ANN can be tackled with a sublinear sample of the stream, storing only $\mathcal{O}(n^{1-\eta})$ points and using $\mathcal{O}(n^{1+\rho-\eta})$ space, with turnstile support and batch-query capability. For KDE in the sliding-window model, the authors fuse RACE with Exponential Histograms to obtain a sublinear sketch of size $\mathcal{O}\left(RW \frac{1}{\sqrt{1+ε}-1} \log^2 N\right)$ that explicitly handles data expiration and supports batches, with empirical results showing close-to-baseline accuracy. Across real and synthetic datasets, the proposed sketches achieve lightweight representations and consistently low error, highlighting practical viability for real-time analytics in large-scale streaming systems.

Abstract

Approximate Nearest Neighbor (ANN) search and Approximate Kernel Density Estimation (A-KDE) are fundamental problems at the core of modern machine learning, with broad applications in data analysis, information systems, and large-scale decision making. In massive and dynamic data streams, a central challenge is to design compact sketches that preserve essential structural properties of the data while enabling efficient queries. In this work, we develop new sketching algorithms that achieve sublinear space and query time guarantees for both ANN and A-KDE for a dynamic stream of data. For ANN in the streaming model, under natural assumptions, we design a sublinear sketch that requires only $\mathcal{O}(n^{1+ρ-η})$ memory by storing only a sublinear ($n^{-η}$) fraction of the total inputs, where $ρ$ is a parameter of the LSH family, and $0<η<1$. Our method supports sublinear query time, batch queries, and extends to the more general Turnstile model. While earlier works have focused on Exact NN, this is the first result on ANN that achieves near-optimal trade-offs between memory size and approximation error. Next, for A-KDE in the Sliding-Window model, we propose a sketch of size $\mathcal{O}\left(RW \cdot \frac{1}{\sqrt{1+ε} - 1} \log^2 N\right)$, where $R$ is the number of sketch rows, $W$ is the LSH range, $N$ is the window size, and $ε$ is the approximation error. This, to the best of our knowledge, is the first theoretical sublinear sketch guarantee for A-KDE in the Sliding-Window model. We complement our theoretical results with experiments on various real-world datasets, which show that the proposed sketches are lightweight and achieve consistently low error in practice.

Figures (11)

• Figure 1: Insert mechanism of Algorithm \ref{['NN_algo']} where, with probability $n^{-\eta}$, we store point ${\bm{x}}_t$ into the hash table using $L$ independent hash functions $g_i({\bm{x}})$.
• Figure 2: Query mechanism of Algorithm \ref{['NN_algo']} which retrieves the closest point to the query vector ${\bm{q}}$ from a candidate set${\mathbb{S}}$ constructed as the union of all hash bucket collision sets,i.e. ${\mathbb{S}} = \bigcup_{i=1}^{L} s_i$, where each $s_i$ denotes the set of points that collide with ${\bm{q}}$ in the $i^{\text{th}}$ hash table.
• Figure 3: Update mechanism of Algorithm \ref{['KDE_algo']}, where $y_i$ is the output of the LSH function $i$ on $x_t$. For the $i^{th}$ row, we add 1 to the EH at index $y_i$
• Figure 4: Query mechanism of Algorithm \ref{['KDE_algo']} where $z_i$ is the output of the LSH function $i$ on $q$. For the $i^{th}$ row, $c_i$ is the count estimate of the EH at index $z_i$.
• Figure 5: Memory requirements scale with stream size $N$ for fixed $\epsilon=0.5$ for the sift1m dataset.

Theorems & Definitions (29)

• Definition 2.1
• Theorem 2.2: har2012approximate
• Theorem 2.3: ACE estimator
• Theorem 2.4
• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Lemma 3.3
• proof