SWING: Unlocking Implicit Graph Representations for Graph Random Features

TL;DR

SWING tackles the challenge of computing Graph Random Features on implicit graphs by replacing discrete node-walks with continuous space-walks in a $d$-dimensional embedding, enabling linear-time kernel actions without materializing edges. It combines Gumbel-softmax relaxations, random feature mappings, and Fourier analysis to approximate the graph kernel $\mathbf{K}_{\boldsymbol{\alpha}}(\mathbf{W})$ via a low-rank factorization $\mathbf{K}_{1}\mathbf{K}_{2}^{\top}$ and precomputed components, achieving $O(NTm)$ time. The framework provides systematic construction of maps $\phi_{f}$ and $\psi_{g}$, supports radial-basis and step-function kernels (including $\epsilon$-neighborhood graphs), and delivers substantial speedups with accurate kernel approximations across synthetic and real-world tasks. Empirically, SWING yields competitive or better performance than GRFs on downstream tasks like vertex normals and Vision Transformer attention, while reducing computational overhead, underscoring its practical impact for large-scale, implicitly defined graphs.

Abstract

We propose SWING: Space Walks for Implicit Network Graphs, a new class of algorithms for computations involving Graph Random Features on graphs given by implicit representations (i-graphs), where edge-weights are defined as bi-variate functions of feature vectors in the corresponding nodes. Those classes of graphs include several prominent examples, such as: $ε$-neighborhood graphs, used on regular basis in machine learning. Rather than conducting walks on graphs' nodes, those methods rely on walks in continuous spaces, in which those graphs are embedded. To accurately and efficiently approximate original combinatorial calculations, SWING applies customized Gumbel-softmax sampling mechanism with linearized kernels, obtained via random features coupled with importance sampling techniques. This algorithm is of its own interest. SWING relies on the deep connection between implicitly defined graphs and Fourier analysis, presented in this paper. SWING is accelerator-friendly and does not require input graph materialization. We provide detailed analysis of SWING and complement it with thorough experiments on different classes of i-graphs.

Figures (11)

• Figure 1: Pictorial visualization of the SWING mechanism on the example of the i-graph obtained from the point cloud. The scheme above the red line shows an approach based on regular Graph Random Features: an explicit graph representation with nodes and weighted edges is constructed and used to conduct walks (here depicted via blue arrows) on the nodes of the constructed graph. SWING bypasses this step by conducting those random walks in the continuous $d$-dimensional space, where i-graph's nodes are embedded (see: scheme to which red arrow is pointing). Each transition in such a walk (see: orange arrow) is obtained as a convex sum of the vectors joining current location of the walker with the locations of the nodes of the graph. The coefficients of that convex sum are proportional to the exponentiated sums of the weight-logarithms and Gumbel variables. That turns out to provide differentiable relaxation of the discrete random walk.
• Figure 2: Frobenius norm error on synthetic point clouds of increasing size for the $p$-step random walk kernel.
• Figure 3: Trade-off between approximation error (FNE) and computation time for the $p$-step random walk kernel on synthetic point clouds of varying size.
• Figure 4: Cosine Similarities for some larger meshes. SWING performs at-par with GRFs.
• Figure 5: Kernel construction time on large meshes, measured on CPU. SWING demonstrates superior scalability compared to GRF.

Theorems & Definitions (3)

• Proposition 2.1
• Proposition 1.1
• proof