Manifold Random Features

TL;DR

This work introduces Manifold Random Features (MRFs), a framework to approximate bi-variate functions on manifolds by learning continuous feature fields through discretization and Graph Random Features (GRFs). By training a neural surrogate to realize a bilinear form $F(x,y)\approx \int g(x,\omega)g(y,\omega)d\omega$, MRFs produce a positive, bounded feature map that enables efficient kernel-vector computations on non-Euclidean domains. The authors establish a deep connection between discrete GRFs and continuous kernel representations, derive RF constructions for diffusion/heat and Gaussian kernels, and show that Gaussian RFs emerge naturally from grid-based GRFs. Extensive experiments on Euclidean spaces and 2D surfaces (sphere, ellipsoid, Möbius strip, torus) demonstrate high-fidelity kernel approximations and substantial speedups, with mesh-interpolation tasks illustrating practical scalability. Overall, MRFs offer a scalable path for kernel methods on manifolds, with potential impact on ML applications involving non-Euclidean data.

Abstract

We present a new paradigm for creating random features to approximate bi-variate functions (in particular, kernels) defined on general manifolds. This new mechanism of Manifold Random Features (MRFs) leverages discretization of the manifold and the recently introduced technique of Graph Random Features (GRFs) to learn continuous fields on manifolds. Those fields are used to find continuous approximation mechanisms that otherwise, in general scenarios, cannot be derived analytically. MRFs provide positive and bounded features, a key property for accurate, low-variance approximation. We show deep asymptotic connection between GRFs, defined on discrete graph objects, and continuous random features used for regular kernels. As a by-product of our method, we re-discover recently introduced mechanism of Gaussian kernel approximation applied in particular to improve linear-attention Transformers, considering simple random walks on graphs and by-passing original complex mathematical computations. We complement our algorithm with a rigorous theoretical analysis and verify in thorough experimental studies.

Figures (12)

• Figure 1: An illustration of the heat kernel $F:\mathcal{M} \times \mathcal{M} \rightarrow \mathbb{R}$ defined on the saddle-like surface $\mathcal{M}$ with a distinguished point $\mathbf{x} \in \mathcal{M}$ marked green. The values $F(\mathbf{x}, \mathbf{y})$ for different $\mathbf{y} \in \mathcal{M}$ are color-coded (different shades of red), with a couple of samples $\mathbf{y}$ highlighted as blue dots. For a general manifold, $F$ is given by a system of partial differential equations and neither it nor the corresponding RF-mechanism are given by the closed-form expressions.
• Figure 2: Four grid-graphs of sizes: $4 \times 4$, $8 \times 8$, $16 \times 16$, $32 \times 32$ with the distinguished vertex $\mathbf{z}$ and its corresponding signature vectors obtained by applying GRF algorithm grfs-1. The signature vectors are represented by color-coding different vertices (with more intense shades corresponding to large values and the most intense used to color vertex $\mathbf{z}$). Next to the graphs, those signature vectors are also represented by color-coding unit-squares. As the resolution of the grid goes to infinity, those representations converge to the continuous object, namely function $g_{i}$ from Eq. \ref{['eq:main_rfs']}.
• Figure 3: Empirical relative mean squared errors (MSEs) of the approximations of $g$-functions and true Gaussian kernel values with signature vectors and kernels induced by them for $d \in \{2,4,...,32\}$ and $n=5,15,...,105$. Relative MSE of the approximation of vector $\mathbf{y}$ with random vector $\mathbf{x}$ is defined as $\mathbb{E}[||\mathbf{x}-\mathbf{y}||^2_2/||\mathbf{y}||^2_2]$. We use $s=30$ repetitions for each $n$ value (the standard deviations are very small compared to the mean so not visible on the figure).
• Figure 4: Qualitative field and heat kernel comparisons for a sphere (left), an ellipsoid (middle), and a Möbius strip (right). Example validation start (top row): predicted $g_\theta(\mathbf{x},\cdot)$, ground truth $\phi_f(x)[\cdot]$, and residual $g_\theta(\mathbf{x},\cdot)-\phi_f(x)[\cdot]$ on the geometry. Reference graph heat kernel (bottom row) $\mathrm{K}^{\mathrm{heat}}(\mathbf{x},\cdot)$ compared to Frobenius-aligned induced kernel $\widetilde{\mathrm{K}}_{\theta}(\mathbf{x},\cdot)=\widetilde{\mathrm{K}}_{\mathrm{NN}}(\mathbf{x},\cdot)$, and their difference, for a representative start point.
• Figure 5: MRFs vs full kernel (FK) construction for vertex normal prediction; preprocessing and interpolation times are denoted by '-P' and '-I', respectively.

Theorems & Definitions (8)

• Theorem 1.1: Convergence of Discrete to Continuous Laplacian
• proof
• Theorem 1.2: Convergence of Discrete Diffusion Kernel to Periodized Gaussian
• proof
• Theorem 1.3: RBF Kernel Factorization
• proof
• Lemma 1.4
• proof