Gransformer: Transformer-based Graph Generation

TL;DR

Gransformer presents an autoregressive graph-generation framework built on a single masked transformer encoder. It injects graph structure through a graph-based familiarity measure and a graph-aware positional encoding, and uses a shared MADE to model dependent edge generation within one forward pass, while BFS ordering prevents isolated nodes. Empirical results on real and synthetic graphs show competitive performance against state-of-the-art autoregressive methods, with ablations highlighting the benefits of graph-aware encoding and dependent edge modeling. The approach offers a scalable, structure-aware alternative for graph generation with potential extensions to more complex graph types and labeling schemes.

Abstract

Transformers have become widely used in various tasks, such as natural language processing and machine vision. This paper proposes Gransformer, an algorithm based on Transformer for generating graphs. We modify the Transformer encoder to exploit the structural information of the given graph. The attention mechanism is adapted to consider the presence or absence of edges between each pair of nodes. We also introduce a graph-based familiarity measure between node pairs that applies to both the attention and the positional encoding. This measure of familiarity is based on message-passing algorithms and contains structural information about the graph. Also, this measure is autoregressive, which allows our model to acquire the necessary conditional probabilities in a single forward pass. In the output layer, we also use a masked autoencoder for density estimation to efficiently model the sequential generation of dependent edges connected to each node. In addition, we propose a technique to prevent the model from generating isolated nodes without connection to preceding nodes by using BFS node orderings. We evaluate this method using synthetic and real-world datasets and compare it with related ones, including recurrent models and graph convolutional networks. Experimental results show that the proposed method performs comparatively to these methods.

Figures (4)

• Figure 1: The schema of the self-attention layer of the transformer encoder. As illustrated in this figure, for each $1 \le i \le n$, the value $v_i^{(t+1)}$ is computed from values $v_j^{(t)}$ for all $1 \le j \le n$. From this point of view, this layer is similar to a graph convolutional layer on a fully connected graph. When we are using an autoregressive mask, $v_i^{(t+1)}$ would be computed from values $v_j^{(t)}$ for all $1 \le j \le i$, and this operation would be similar to a graph convolutional layer on a fully connected directed acyclic graph.
• Figure 2: Schema of the Gransformer model. Notice that the autoregressive (MADE) network used at the output layer is shared between all the $n$ output vectors of the transformer encoder.
• Figure 3: Here is an example of a MADE network appearing at the output layer. Numbers written on the network units and inputs are the tags. Masked connections have been removed in this figure. The $i$-th output of the transformer encoder contains the information from the $i-1$ preceding nodes of the graph. Therefore, it is tagged by zero to be used by all the output units. The target vector contains the information on the edges connecting the $i$-th node to the preceding nodes. The network masks guarantee that the $j$-th output unit uses only the first $j-1$ elements of the target vector. It is worth noting that only the first $i-1$ elements of $\bar{L}_i$ are of our interest, and the rest will be treated as zero. To keep the figure simple and sparse, we ignored drawing direct links from the input layer to the output layer.
• Figure 4: Some original samples and generated graphs by different methods, for each dataset.

Theorems & Definitions (2)

• theorem 1
• proof