Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries

Haitz Sáez de Ocáriz Borde, Anastasis Kratsios

TL;DR

Neural Snowflakes introduce a trainable fractal-like metric on $\mathbb{R}^d$ that, when combined with a simple MLP encoder, achieves universal graph embedding for finite weighted graphs and enables differentiable latent-geometry learning for latent graph inference. The authors prove that pure Riemannian representations cannot be universal, but neural snowflakes overcome this limitation, providing both isometric and distortion-tolerant guarantees with finite, dimension-independent parameter counts. Empirically, the approach matches or surpasses state-of-the-art latent graph inference methods on standard benchmarks and synthetic tasks, doing so without relying on random search over geometries. Together, the theory and experiments establish differentiable, adaptable latent geometries as a principled foundation for scalable graph structure learning.

Abstract

The inductive bias of a graph neural network (GNN) is largely encoded in its specified graph. Latent graph inference relies on latent geometric representations to dynamically rewire or infer a GNN's graph to maximize the GNN's predictive downstream performance, but it lacks solid theoretical foundations in terms of embedding-based representation guarantees. This paper addresses this issue by introducing a trainable deep learning architecture, coined neural snowflake, that can adaptively implement fractal-like metrics on $\mathbb{R}^d$. We prove that any given finite weights graph can be isometrically embedded by a standard MLP encoder. Furthermore, when the latent graph can be represented in the feature space of a sufficiently regular kernel, we show that the combined neural snowflake and MLP encoder do not succumb to the curse of dimensionality by using only a low-degree polynomial number of parameters in the number of nodes. This implementation enables a low-dimensional isometric embedding of the latent graph. We conduct synthetic experiments to demonstrate the superior metric learning capabilities of neural snowflakes when compared to more familiar spaces like Euclidean space. Additionally, we carry out latent graph inference experiments on graph benchmarks. Consistently, the neural snowflake model achieves predictive performance that either matches or surpasses that of the state-of-the-art latent graph inference models. Importantly, this performance improvement is achieved without requiring random search for optimal latent geometry. Instead, the neural snowflake model achieves this enhancement in a differentiable manner.

Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries

TL;DR

Neural Snowflakes introduce a trainable fractal-like metric on that, when combined with a simple MLP encoder, achieves universal graph embedding for finite weighted graphs and enables differentiable latent-geometry learning for latent graph inference. The authors prove that pure Riemannian representations cannot be universal, but neural snowflakes overcome this limitation, providing both isometric and distortion-tolerant guarantees with finite, dimension-independent parameter counts. Empirically, the approach matches or surpasses state-of-the-art latent graph inference methods on standard benchmarks and synthetic tasks, doing so without relying on random search over geometries. Together, the theory and experiments establish differentiable, adaptable latent geometries as a principled foundation for scalable graph structure learning.

Abstract

The inductive bias of a graph neural network (GNN) is largely encoded in its specified graph. Latent graph inference relies on latent geometric representations to dynamically rewire or infer a GNN's graph to maximize the GNN's predictive downstream performance, but it lacks solid theoretical foundations in terms of embedding-based representation guarantees. This paper addresses this issue by introducing a trainable deep learning architecture, coined neural snowflake, that can adaptively implement fractal-like metrics on . We prove that any given finite weights graph can be isometrically embedded by a standard MLP encoder. Furthermore, when the latent graph can be represented in the feature space of a sufficiently regular kernel, we show that the combined neural snowflake and MLP encoder do not succumb to the curse of dimensionality by using only a low-degree polynomial number of parameters in the number of nodes. This implementation enables a low-dimensional isometric embedding of the latent graph. We conduct synthetic experiments to demonstrate the superior metric learning capabilities of neural snowflakes when compared to more familiar spaces like Euclidean space. Additionally, we carry out latent graph inference experiments on graph benchmarks. Consistently, the neural snowflake model achieves predictive performance that either matches or surpasses that of the state-of-the-art latent graph inference models. Importantly, this performance improvement is achieved without requiring random search for optimal latent geometry. Instead, the neural snowflake model achieves this enhancement in a differentiable manner.
Paper Structure (28 sections, 16 theorems, 60 equations, 2 figures, 9 tables, 3 algorithms)

This paper contains 28 sections, 16 theorems, 60 equations, 2 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Adaptive Geometries via Neural Snowflakes
  4. Neural Snowflakes
  5. Inferability Guarantees
  6. Universal Graph Embedding
  7. Universal Riemannian Representation Is Impossible
  8. Universal Representation Is Possible With Neural Snowflake
  9. Isometric Representation Guarantees - By Small Neural Snowflakes
  10. Representation Guarantees Leveraging Distortion
  11. Experimental Results
  12. Conclusion
  13. Additional Background
  14. Metric Spaces
  15. MLPs with ReLU Activation Function
  16. ...and 13 more sections

Key Result

Proposition 1

Let $f:[0,\infty)\rightarrow [0,\infty)$ be a continuous, concave, monotonically increasing function with $f(0)=0$, and let $(X,d)$ be a metric space; then, $d_f:X\times X\rightarrow \mathbb{R}$ is a metric on $X$$d_f(x,z) \stackrel{\hbox{\upshape\tiny def.}}{=} f(d(x,z)) ,$ for any $x,z\in X$.

Figures (2)

  • Figure 1: Explanation of Proposition \ref{['prop:Embedding_Impossible']}: The Graph of Proposition \ref{['prop:Embedding_Impossible']} cannot be isometrically embedded into any complete and connected (smooth) Riemannian manifold. Briefly, the issue is that any isometric embedding into such a Riemannian manifold must exhibit a pair of geodesics one of which travels from the embeddings of node $C$ to node $A$, while passing through the embedding of node $D$; and likewise, the other of which travels from the embedding of node $C$ to node $B$ and again passes through the embedding of node $D$. However, this would violate the local uniqueness of geodesics in such a Riemannian manifold, around the embedding of node $D$ (implied by the Picard-Lindelöf theorem for ODEs); thus no such embedding can exist.
  • Figure 2: Training losses for synthetic graph embedding experiments. We compare using Euclidean space for encoding the weighted graphs to using snowflake quasi-metric spaces.

Theorems & Definitions (31)

  • Example 1: xia2009geodesic
  • Proposition 1: Snowflakes are Metric Spaces - WeaverLipschitzAlgebras_2ed_2018
  • Definition 1: Universal Graph Embedding
  • Theorem 1: Generic Graph Reconstruction via Universal Graph Inference Models
  • Proposition 2: Riemannian Representation Spaces are Too Rigid to be Universal
  • Theorem 2: Universal Graph Embedding
  • Theorem 3: Neural Snowflakes $\&$ MLPs Are More Powerful For Representation Learning Than MLPs
  • Example 2
  • Example 3
  • Theorem 4: Quantitative Embedding Guarantees for Bounded Metric Geometries
  • ...and 21 more