Improved Random Features for Dot Product Kernels

TL;DR

This work tackles the scalability bottleneck of dot-product kernel methods by introducing complex-valued random features for polynomial sketches and a variance-driven optimization (Optimized Maclaurin) that allocates random features across polynomial degrees. It provides closed-form variance formulas for both complex polynomial sketches and TensorSRHT, enabling principled comparisons and practical data-driven feature construction applicable to general dot-product kernels, including the Gaussian kernel. The authors demonstrate substantial variance reduction and improved downstream GP performance across diverse datasets, with extensive experiments and a GPU-accelerated software package. The results advance the practical utility of dot-product kernels in large-scale learning by combining theoretical variance analysis with a tunable, data-driven feature allocation strategy.

Abstract

Dot product kernels, such as polynomial and exponential (softmax) kernels, are among the most widely used kernels in machine learning, as they enable modeling the interactions between input features, which is crucial in applications like computer vision, natural language processing, and recommender systems. We make several novel contributions for improving the efficiency of random feature approximations for dot product kernels, to make these kernels more useful in large scale learning. First, we present a generalization of existing random feature approximations for polynomial kernels, such as Rademacher and Gaussian sketches and TensorSRHT, using complex-valued random features. We show empirically that the use of complex features can significantly reduce the variances of these approximations. Second, we provide a theoretical analysis for understanding the factors affecting the efficiency of various random feature approximations, by deriving closed-form expressions for their variances. These variance formulas elucidate conditions under which certain approximations (e.g., TensorSRHT) achieve lower variances than others (e.g., Rademacher sketches), and conditions under which the use of complex features leads to lower variances than real features. Third, by using these variance formulas, which can be evaluated in practice, we develop a data-driven optimization approach to improve random feature approximations for general dot product kernels, which is also applicable to the Gaussian kernel. We describe the improvements brought by these contributions with extensive experiments on a variety of tasks and datasets.

Figures (19)

• Figure 1: Multiplying each element of a random vector $\boldsymbol{z}_{i,j}$ in \ref{['example:complex-rademacher-sketch']} by $\exp(\mathrm{i}\mkern1mu \frac{\pi}{4})$ corresponds to a counter-clockwise rotation of that element by 45 degrees on the complex plane. The support of the resulting elements is $\{1, -1, \mathrm{i}\mkern1mu, -\mathrm{i}\mkern1mu \}$ and the construction of \ref{['example:complex-rademacher-uniform']} is obtained.
• Figure 2: This plot shows the mean squared error $\mathbb{E}[ | \hat{k}(\boldsymbol{x}, \boldsymbol{y}) - (\boldsymbol{x}^\top \boldsymbol{y})^p |^2 ]$ for different values of the degree $p$ as well as the mean squared error ratio of the complex sketches over their real analogues. We sample 1,000 independent vector pairs $(\boldsymbol{x}, \boldsymbol{y}) \in \mathbb{R}^d \times \mathbb{R}^d$ with $\boldsymbol{x}=|\tilde{\boldsymbol{x}}| / \| \tilde{\boldsymbol{x}} \|, \boldsymbol{y}=|\tilde{\boldsymbol{y}}| / \| \tilde{\boldsymbol{y}} \|$ and $\tilde{\boldsymbol{x}}, \tilde{\boldsymbol{y}} \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I}_d)$. The mean is then taken over 1,000 independent constructions of the approximate kernel $\hat{k}(\boldsymbol{x}, \boldsymbol{y})$, and over every input pair $(\boldsymbol{x}, \boldsymbol{y})$.
• Figure 3: Empirical cumulative distribution of pairwise ratios Var(Compl. TensorSRHT) / Var(Real TensorSRHT) on a subsample (1000 samples) of four different datasets (EEG, CIFAR10 ResNet34 features, MNIST, Gisette) with unit-normalized data where $D=d$. The datasets are not zero-centered and therefore entirely positive.
• Figure 4: Numerical illustration of Section \ref{['sec:illustration-objective-optim']}. The left two figures are box plots for the Gaussian kernel (i), and the right two figures are those for the polynomial kernel (ii). The top figures show the variance terms (a), and the bottom figures show the bias terms (b). See Section \ref{['sec:illustration-objective-optim']} for details.
• Figure 5: One-dimensional GP regression experiment in Section \ref{['sec:GP-toy-experiment']}. The top row shows the results of random Fourier features (Gaussian RFF), and the bottom row those of the optimized Maclaurin approach. The left, middle, and right columns correspond to the ground-truth sinc functions with frequencies of $5$, $2$, and $0.5$, respectively. The values of $l$ and $\sigma^2$ are the kernel hyperparameters obtained by maximizing the log likelihood of training data in the full GP (i.e., without approximation). Dashed black curves represent approximate GP posterior mean functions; black curves represent the posterior means plus and minus 2 times approximate posterior standard deviations; black curves represent the posterior mean functions of the full GP; and the shaded areas are the full GP posterior means plus and minus 2 times the full GP posterior deviations.

Theorems & Definitions (19)

• Example 1
• Example 2: Complex Rademacher Sketch
• Example 3: Complex Gaussian Sketch
• Example 4
• Proposition 1
• Theorem 2
• Theorem 3
• Proposition 4
• Theorem 5: Variance of Real TensorSRHT
• Remark 6