Subgraphormer: Unifying Subgraph GNNs and Graph Transformers via Graph Products

TL;DR

Subgraphormer addresses expressivity gaps in graph learning by unifying Subgraph GNNs with Graph Transformers through a product-graph lens. It shows Subgraph GNNs can be implemented as MPNNs on a Cartesian product, enabling an attention-driven SAB and a novel product-graph PE with efficient eigendecomposition. The method yields state-of-the-art or competitive results across ZINC, OGB, and long-range peptide benchmarks, and its stochastic sampling variant scales to larger graphs while preserving performance. This work offers a practical path to combining local subgraph messages with global transformer-style attention, with potential extensions to higher-order (k-tuple) graph representations.

Abstract

In the realm of Graph Neural Networks (GNNs), two exciting research directions have recently emerged: Subgraph GNNs and Graph Transformers. In this paper, we propose an architecture that integrates both approaches, dubbed Subgraphormer, which combines the enhanced expressive power, message-passing mechanisms, and aggregation schemes from Subgraph GNNs with attention and positional encodings, arguably the most important components in Graph Transformers. Our method is based on an intriguing new connection we reveal between Subgraph GNNs and product graphs, suggesting that Subgraph GNNs can be formulated as Message Passing Neural Networks (MPNNs) operating on a product of the graph with itself. We use this formulation to design our architecture: first, we devise an attention mechanism based on the connectivity of the product graph. Following this, we propose a novel and efficient positional encoding scheme for Subgraph GNNs, which we derive as a positional encoding for the product graph. Our experimental results demonstrate significant performance improvements over both Subgraph GNNs and Graph Transformers on a wide range of datasets.

Figures (8)

• Figure 1: An example of generating subgraphs from the original graph. We denote by $v$ the index used for the node dimension, and by $s$ the one employed for the subgraph dimension. Each subgraph is generated by marking a single node in the original graph, with the marked node represented in black. We refer to the marked node as the root of the corresponding subgraph.
• Figure 2: An overview of Subgraphormer. Given an input graph, we first construct the product graph and compute a subgraph-specific positional encoding. Then, we apply a stacking of Subgraph Attention Blocks, followed by a pooling layer to obtain a graph representation. A more comprehensive depiction of this figure can be found in \ref{['fig: architecture_full']} of \ref{['app: Subgraphormer Figure']}.
• Figure 3: A deep overview of Subgraphormer. Given an input graph, the process begins with the construction of the product graph. This is followed by the computation of product graph PE, illustrated through varying colors on the nodes. Node-Marking is then implemented, depicted by black exes () on diagonal nodes. The process continues with the application of $K$ Subgraph Attention Blocks (SABs), characterized by three distinct connectivities: Internal, External, and Point. The final stage involves the pooling layer, which initially aggregates data across the node dimension to form subgraph representations, and subsequently across the subgraph dimension.
• Figure :
• Figure :

Theorems & Definitions (39)

• Proposition 3.1: GNN-SSWL+ as an MPNN on the product graph
• Proposition 3.2: Internal and External Connectivities give rise to the Cartesian Product Graph
• Proposition 4.1: Product Graph eigendecomposition
• Definition 1.1: Cartesian Product Graph
• Corollary 1.1
• Proposition 1.1: Internal and External Connectivities give rise to the Cartesian Product Graph
• Proposition 1.1: Product Graph eigendecomposition
• Proposition 1.1: Product Graph PE Complexity
• Corollary 1.2: Adjacency of Cartesian Product Graph
• Corollary 2.1: Adjacency of Cartesian Product Graph