Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Representation Learning with Diffusion Generative Models

Daniel Wesego

TL;DR

The paper tackles graph representation learning by leveraging discrete diffusion processes to produce informative graph embeddings. It introduces the Discrete Diffusion Graph Autoencoder (DDGAE), which combines a GCN encoder with a UNet-style diffusion decoder conditioned on the encoder latent, and extracts the final representation as the concatenation $\mathbf{z} = [\mathbf{z}_{enc}; \mathbf{z}_{dec}^{(t)}]$. Training optimizes the discrete diffusion objective $L_\lambda = L_{VB} + \lambda \mathbb{E}_{q(\boldsymbol{x}_0)} \mathbb{E}_{q(\boldsymbol{x}_t|\boldsymbol{x}_0)} [-\log \tilde{p}_\theta(\boldsymbol{x}_0|\boldsymbol{x}_t)]$, enabling unsupervised learning with single-pass inference at test time. Empirical results on PROTEINS and IMDB-BINARY show DDGAE achieves state-of-the-art performance among baselines, illustrating the potential of discrete diffusion autoencoders for graph representations, while noting limitations in computational cost and scope to static graphs and suggesting future work on efficient diffusion schedules and broader graph types.

Abstract

Diffusion models have established themselves as state-of-the-art generative models across various data modalities, including images and videos, due to their ability to accurately approximate complex data distributions. Unlike traditional generative approaches such as VAEs and GANs, diffusion models employ a progressive denoising process that transforms noise into meaningful data over multiple iterative steps. This gradual approach enhances their expressiveness and generation quality. Not only that, diffusion models have also been shown to extract meaningful representations from data while learning to generate samples. Despite their success, the application of diffusion models to graph-structured data remains relatively unexplored, primarily due to the discrete nature of graphs, which necessitates discrete diffusion processes distinct from the continuous methods used in other domains. In this work, we leverage the representational capabilities of diffusion models to learn meaningful embeddings for graph data. By training a discrete diffusion model within an autoencoder framework, we enable both effective autoencoding and representation learning tailored to the unique characteristics of graph-structured data. We extract the representation from the combination of the encoder's output and the decoder's first time step hidden embedding. Our approach demonstrates the potential of discrete diffusion models to be used for graph representation learning. The code can be found at https://github.com/DanielMitiku/Graph-Representation-Learning-with-Diffusion-Generative-Models

Graph Representation Learning with Diffusion Generative Models

TL;DR

The paper tackles graph representation learning by leveraging discrete diffusion processes to produce informative graph embeddings. It introduces the Discrete Diffusion Graph Autoencoder (DDGAE), which combines a GCN encoder with a UNet-style diffusion decoder conditioned on the encoder latent, and extracts the final representation as the concatenation . Training optimizes the discrete diffusion objective , enabling unsupervised learning with single-pass inference at test time. Empirical results on PROTEINS and IMDB-BINARY show DDGAE achieves state-of-the-art performance among baselines, illustrating the potential of discrete diffusion autoencoders for graph representations, while noting limitations in computational cost and scope to static graphs and suggesting future work on efficient diffusion schedules and broader graph types.

Abstract

Diffusion models have established themselves as state-of-the-art generative models across various data modalities, including images and videos, due to their ability to accurately approximate complex data distributions. Unlike traditional generative approaches such as VAEs and GANs, diffusion models employ a progressive denoising process that transforms noise into meaningful data over multiple iterative steps. This gradual approach enhances their expressiveness and generation quality. Not only that, diffusion models have also been shown to extract meaningful representations from data while learning to generate samples. Despite their success, the application of diffusion models to graph-structured data remains relatively unexplored, primarily due to the discrete nature of graphs, which necessitates discrete diffusion processes distinct from the continuous methods used in other domains. In this work, we leverage the representational capabilities of diffusion models to learn meaningful embeddings for graph data. By training a discrete diffusion model within an autoencoder framework, we enable both effective autoencoding and representation learning tailored to the unique characteristics of graph-structured data. We extract the representation from the combination of the encoder's output and the decoder's first time step hidden embedding. Our approach demonstrates the potential of discrete diffusion models to be used for graph representation learning. The code can be found at https://github.com/DanielMitiku/Graph-Representation-Learning-with-Diffusion-Generative-Models
Paper Structure (15 sections, 3 equations, 1 figure, 1 table)

This paper contains 15 sections, 3 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Diffusion Models
  4. Discrete Diffusion Models for Graphs
  5. Graph Representation Learning
  6. Methodology
  7. Discrete Diffusion Framework
  8. Discrete Diffusion Graph Autoencoder
  9. Experiments
  10. Dataset
  11. Baselines
  12. Model Architecture and Training
  13. Evaluation
  14. Results
  15. Conclusion and Discussion

Figures (1)

  • Figure 1: Discrete Diffusion Graph AutoEncoder (DDGAE) embedding extraction: The trained encoder extracts features, which are concatenated with the intermediate output of the trained diffusion decoder as the final embedding $\mathbf{z}$ that will be used for different downstream tasks.