A Dynamic Low-Rank Fast Gaussian Transform

TL;DR

This work tackles the challenge of dynamically maintaining fast Gaussian transforms when data points evolve. It develops a deterministic, dynamic FGT data structure that supports Insert/Delete of sources and KDE-Query with additive error $\varepsilon$ in time polylogarithmic in $n/\varepsilon$ when sources lie in a low-dimensional subspace of dimension $w$, while preserving quasi-linear complexity for matrix-vector queries $\mathsf{K}q$. The core idea is to decouple source–target interactions using truncated Hermite and Taylor expansions, storing and updating coefficients that capture near-field interactions, and extending naturally to fast-decaying kernels. The framework also covers projections to low-dimensional subspaces and handles dynamic changes to the subspace rank, offering sublinear updates in practical, adaptive settings. Overall, the results enable efficient, deterministic dynamic kernel-density estimation and kernel-based computations in evolving datasets, with broad relevance to online regression, KDE, and kernel methods in ML.

Abstract

The \emph{Fast Gaussian Transform} (FGT) enables subquadratic-time multiplication of an $n\times n$ Gaussian kernel matrix $\mathsf{K}_{i,j}= \exp ( - \| x_i - x_j \|_2^2 ) $ with an arbitrary vector $h \in \mathbb{R}^n$, where $x_1,\dots, x_n \in \mathbb{R}^d$ are a set of \emph{fixed} source points. This kernel plays a central role in machine learning and random feature maps. Nevertheless, in most modern data analysis applications, datasets are dynamically changing (yet often have low rank), and recomputing the FGT from scratch in (kernel-based) algorithms incurs a major computational overhead ($\gtrsim n$ time for a single source update $\in \mathbb{R}^d$). These applications motivate a \emph{dynamic FGT} algorithm, which maintains a dynamic set of sources under \emph{kernel-density estimation} (KDE) queries in \emph{sublinear time} while retaining Mat-Vec multiplication accuracy and speed. Assuming the dynamic data-points $x_i$ lie in a (possibly changing) $k$-dimensional subspace ($k\leq d$), our main result is an efficient dynamic FGT algorithm, supporting the following operations in $\log^{O(k)}(n/\varepsilon)$ time: (1) Adding or deleting a source point, and (2) Estimating the ``kernel-density'' of a query point with respect to sources with $\varepsilon$ additive accuracy. The core of the algorithm is a dynamic data structure for maintaining the \emph{projected} ``interaction rank'' between source and target boxes, decoupled into finite truncation of Taylor and Hermite expansions.

Figures (3)

• Figure 1: An illustration of the source-target boxing our data structure maintains in high dimensional space, using the "hybrid" of Taylor-Hermite expansions.
• Figure 2: An illustration of inserting two source points with corresponding interactions to the data structure.
• Figure 3: An illustration of deleting a source point from the data structure.

Theorems & Definitions (34)

• Theorem 1.1: Dynamic Low-Rank FGT, Informal version of Theorem \ref{['thm:fmm_online_low_rank']}
• Remark 2.1
• Lemma 2.2: Hermite projection lemma in low-dimensional space, informal version of Lemma \ref{['lem:proj_expansion']}
• Definition A.1: One-dimensional Hermite polynomial, hermite1864
• Definition A.2: One-dimensional Hermite function, hermite1864
• Lemma A.4: Cramer's inequality for one-dimensional, h26
• Lemma A.5
• Definition A.6: Multi-dimensional Hermite polynomial, hermite1864
• Definition A.7: Multi-dimensional Hermite function, hermite1864
• Lemma A.8: Cramer's inequality for multi-dimensional case, gs91acss20