Tensor Sketch: Fast and Scalable Polynomial Kernel Approximation

TL;DR

The paper introduces Tensor Sketch to efficiently approximate the polynomial kernel $\kappa(\mathbf x, \mathbf y) = (c + \langle \mathbf x, \mathbf y \rangle)^p$ by combining tensor powers with Count Sketch and FFT-based convolution. It achieves a per-vector time of $O(d + D\log D)$ and total training time $O(n(d + D\log D))$ with $O(1)$ extra space, enabling scalable kernel methods on large, high-dimensional datasets. The authors prove unbiasedness and provide a variance bound $\mathrm{Var}[\langle \mathbf{Cx}^{(p)}, \mathbf{Cy}^{(p)} \rangle] \le (3^p - 1)/D \|\mathbf x\|_2^{2p} \|\mathbf y\|_2^{2p}$, establishing reliable approximation guarantees. The approach has influenced subsequent work and practical implementations (e.g., in scikit-learn), highlighting its significance for fast, scalable kernel-based learning in diverse domains.

Abstract

Approximation of non-linear kernels using random feature maps has become a powerful technique for scaling kernel methods to large datasets. We propose $\textit{Tensor Sketch}$, an efficient random feature map for approximating polynomial kernels. Given $n$ training samples in $\mathbb{R}^d$ Tensor Sketch computes low-dimensional embeddings in $\mathbb{R}^D$ in time $\mathcal{O}\left( n(d+D \log{D}) \right)$ making it well-suited for high-dimensional and large-scale settings. We provide theoretical guarantees on the approximation error, ensuring the fidelity of the resulting kernel function estimates. We also discuss extensions and highlight applications where Tensor Sketch serves as a central computational tool.

Theorems & Definitions (9)

• Lemma 1
• Definition 2
• Definition 3
• Lemma 4
• Definition 5
• Lemma 6
• Lemma 7
• Lemma 8
• Theorem 9