A Quasi-Monte Carlo Data Structure for Smooth Kernel Evaluations

TL;DR

This work advances kernel density estimation for high-dimensional, smooth positive-definite radial kernels by marrying discrepancy theory with randomized space partitioning. The authors develop new data-dependent feature embeddings that exploit well-separated shells to dramatically reduce the discrepancy of kernel matrices, enabling coresets that feed into a data structure with polylogarithmic dependence on $1/\mu$ and linear dependence on $1/\epsilon$. The main contribution is a framework that achieves a $(1±\epsilon)$-approximation with space $\approx n \cdot L \cdot (d\log(n\Phi/(\epsilon\mu)))^{O(t)}/\epsilon$ and query time $\approx L \cdot (d\log(n\Phi/(\epsilon\mu)))^{O(t)}/\epsilon$ for $(L,t)$-smooth PD radial kernels, leveraging a ball-carving hash and feature-embedding-based discrepancy bounds. The approach provides new structural insights into kernel matrices and paves the way for fast, high-dimensional kernel evaluations with potential applications to numerical linear algebra on kernel matrices. It also outlines clear open questions, such as extending to non-smooth kernels like the Gaussian and integrating with existing fast-multipole-type frameworks.

Abstract

In the kernel density estimation (KDE) problem one is given a kernel $K(x, y)$ and a dataset $P$ of points in a Euclidean space, and must prepare a data structure that can quickly answer density queries: given a point $q$, output a $(1+ε)$-approximation to $μ:=\frac1{|P|}\sum_{p\in P} K(p, q)$. The classical approach to KDE is the celebrated fast multipole method of [Greengard and Rokhlin]. The fast multipole method combines a basic space partitioning approach with a multidimensional Taylor expansion, which yields a $\approx \log^d (n/ε)$ query time (exponential in the dimension $d$). A recent line of work initiated by [Charikar and Siminelakis] achieved polynomial dependence on $d$ via a combination of random sampling and randomized space partitioning, with [Backurs et al.] giving an efficient data structure with query time $\approx \mathrm{poly}{\log(1/μ)}/ε^2$ for smooth kernels. Quadratic dependence on $ε$, inherent to the sampling methods, is prohibitively expensive for small $ε$. This issue is addressed by quasi-Monte Carlo methods in numerical analysis. The high level idea in quasi-Monte Carlo methods is to replace random sampling with a discrepancy based approach -- an idea recently applied to coresets for KDE by [Phillips and Tai]. The work of Phillips and Tai gives a space efficient data structure with query complexity $\approx 1/(εμ)$. This is polynomially better in $1/ε$, but exponentially worse in $1/μ$. We achieve the best of both: a data structure with $\approx \mathrm{poly}{\log(1/μ)}/ε$ query time for smooth kernel KDE. Our main insight is a new way to combine discrepancy theory with randomized space partitioning inspired by, but significantly more efficient than, that of the fast multipole methods. We hope that our techniques will find further applications to linear algebra for kernel matrices.

Figures (2)

• Figure 1: Illustration of the well-separated setup. The point $c \in \mathbb{R}^d$ is the center of ball, which may be assumed to be the origin after a translation, and we consider two well-separated shells at radii between $[r_{\min}, r_{\mathrm{in}}]$ and $[r_{\mathrm{out}}, r_{\max}]$ centered at $c$, where $r_{\mathrm{in}} < r_{\mathrm{out}}$. We consider dataset points which lie in the inner shell, with distance between $r_{\min}$ and $r_{\mathrm{in}}$ from $c$, and queries which lie in the outer shell, with distance between $r_{\mathrm{out}}$ and $r_{\max}$ from $c$. This is the case we consider throughout the technical overview; the symmetric case when the dataset is inside a low radius shell and query is outside will be analogous.
• Figure 2: Illustration of ball-carving with balls of radius $O(R)$ when query is at distance at most $3R$ from $c$.

Theorems & Definitions (32)

• Definition 1: Smooth Function BCIS18
• Remark 1
• Theorem 2: Main result (Informal -- see Theorem \ref{['thm:main-thm']})
• Remark 2
• Definition 3: $\gamma_2$-norm of a matrix
• Theorem 4: Banaszczyk's Theorem
• Theorem 5
• Remark 3: Using Smoothness to Relate $\gamma_2(\tilde{A})$ to $\mathsf{K}(P, q) / |P|$
• Theorem 6: Schoenberg's Characterization
• Theorem 7: Haussdorf-Bernstein-Widder