Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Permutation-Invariant Spectral Learning via Dyson Diffusion

Tassilo Schwarz, Cai Dieball, Constantin Kogler, Kevin Lam, Renaud Lambiotte, Arnaud Doucet, Aljaž Godec, George Deligiannidis

TL;DR

The work tackles diffusion on graphs by addressing the >>n!<< adjacency representations through a spectral diffusion approach. By leveraging Dyson Brownian Motion, it decouples spectral dynamics from eigenvectors, enabling accurate graph-spectrum learning with flexible architectures and without data augmentation. The resulting Dyson Diffusion Model provides a tractable training objective, adaptive numerical schemes, and a shooting mechanism to handle singularities, outperforming state-of-the-art baselines on WL-bimodal and brain-graph datasets. The framework generalizes beyond graphs to symmetric matrices and suggests extensions to Laplacian spectra and eigenvector diffusion, broadening diffusion modeling in structured data.

Abstract

Diffusion models are central to generative modeling and have been adapted to graphs by diffusing adjacency matrix representations. The challenge of having up to $n!$ such representations for graphs with $n$ nodes is only partially mitigated by using permutation-equivariant learning architectures. Despite their computational efficiency, existing graph diffusion models struggle to distinguish certain graph families, unless graph data are augmented with ad hoc features. This shortcoming stems from enforcing the inductive bias within the learning architecture. In this work, we leverage random matrix theory to analytically extract the spectral properties of the diffusion process, allowing us to push the inductive bias from the architecture into the dynamics. Building on this, we introduce the Dyson Diffusion Model, which employs Dyson's Brownian Motion to capture the spectral dynamics of an Ornstein-Uhlenbeck process on the adjacency matrix while retaining all non-spectral information. We demonstrate that the Dyson Diffusion Model learns graph spectra accurately and outperforms existing graph diffusion models.

Permutation-Invariant Spectral Learning via Dyson Diffusion

TL;DR

The work tackles diffusion on graphs by addressing the >>n!<< adjacency representations through a spectral diffusion approach. By leveraging Dyson Brownian Motion, it decouples spectral dynamics from eigenvectors, enabling accurate graph-spectrum learning with flexible architectures and without data augmentation. The resulting Dyson Diffusion Model provides a tractable training objective, adaptive numerical schemes, and a shooting mechanism to handle singularities, outperforming state-of-the-art baselines on WL-bimodal and brain-graph datasets. The framework generalizes beyond graphs to symmetric matrices and suggests extensions to Laplacian spectra and eigenvector diffusion, broadening diffusion modeling in structured data.

Abstract

Diffusion models are central to generative modeling and have been adapted to graphs by diffusing adjacency matrix representations. The challenge of having up to such representations for graphs with nodes is only partially mitigated by using permutation-equivariant learning architectures. Despite their computational efficiency, existing graph diffusion models struggle to distinguish certain graph families, unless graph data are augmented with ad hoc features. This shortcoming stems from enforcing the inductive bias within the learning architecture. In this work, we leverage random matrix theory to analytically extract the spectral properties of the diffusion process, allowing us to push the inductive bias from the architecture into the dynamics. Building on this, we introduce the Dyson Diffusion Model, which employs Dyson's Brownian Motion to capture the spectral dynamics of an Ornstein-Uhlenbeck process on the adjacency matrix while retaining all non-spectral information. We demonstrate that the Dyson Diffusion Model learns graph spectra accurately and outperforms existing graph diffusion models.

Paper Structure

This paper contains 47 sections, 7 theorems, 89 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Limitations of Existing Diffusion Models
  3. Dyson Diffusion Model
  4. Dyson Brownian Motion
  5. From the Dyson SDE to a Diffusion Model
  6. Experiments
  7. Methodology
  8. Results
  9. Related Work
  10. Extensions
  11. Conclusion
  12. Making the loss tractable
  13. Spectral Dyson SDE
  14. Dyson SDE.
  15. Invariant Distribution.
  16. ...and 32 more sections

Key Result

Lemma 2.1

For every fixed $n,k \in \mathbb{N}$, all $k$-regular graphs $G \in \mathcal{G}^n$ are WL equivalent. Moreover, every graph $G \in \mathcal{G}^n$ that is WL equivalent to a $k$-regular graph is $k$-regular.

Figures (7)

  • Figure 1: Dyson Diffusion model and its application to graph spectra: A graph on $n$ vertices (a) has up to $n!$ representations as adjacency matrices (b). For an OU-driven diffusion on any adjacency matrix, the permutation-invariant spectrum (c) evolves according to the same SDE \ref{['eq:SDE-spectrum']}. An exemplary path of the $n$, non-intersecting, eigenvalues is shown. The marginals of the invariant density for the $\lambda_k$ are depicted on the far right. The DyDM diffusion model learns the score $s(\lambda, t)$ (highlighted in yellow) to generate spectra via the time-reversed SDE \ref{['eq:dyson-timereversal']}.
  • Figure 2: Struggle of GNN-based and graph-transformer-based models with two WL-equivalent graphs: Graphs A and Graphs B are WL-equivalent, but non-isomorphic. Also physically, they have very different properties, such as different temperature factors (c) and a different cut size. Upon training on a $80 \%$ Graph A and $20\%$ Graph B dataset, state-of-the-art GNN-based (EDP-GNN,GDSS) and graph-transformer-based (ConGress) models learn the WL-equivalence class quickly but fail to generate the underlying distribution among the two graphs, with some even predominantly hallucinating WL-equivalent but non-isomorphic graphs (d).
  • Figure 3: DyDM training
  • Figure 4: Plot of the invariant density of \ref{['eq:SDE-spectrum']} for $d = 2$ and $\alpha = \beta = 1$
  • Figure 5: Forward stepsize controller for Dyson SDE
  • ...and 2 more figures

Theorems & Definitions (13)

  • Lemma 2.1: WL-equivalence of $k$-regular graphs
  • Theorem 3.1: Eigenvalue SDE, dysonBrownianMotionModelEigenvalues1962
  • Theorem 3.2: Eigenvector SDE
  • proof
  • proof
  • Lemma C.1
  • proof
  • Corollary K.1: Mean square error
  • Lemma K.2: MSE of ${\rm est_A}$
  • proof
  • ...and 3 more