Demystifying Higher-Order Graph Neural Networks

TL;DR

This work addresses the challenge of understanding and selecting higher-order graph neural networks (HOGNNs) by introducing a formal taxonomy and a blueprint for constructing HOGNNs. It formalizes a broad class of higher-order graph data models (GDMs) and corresponding MP architectures, along with lifting/lowering operations, to unify disparate HOGNN approaches. The paper surveys and analyzes over 100 HOGNN schemes in terms of expressiveness and computational costs, offering guidance on when to deploy particular HO structures and how to balance power with efficiency. It further discusses limitations and future directions, highlighting the need for scalable HO methods, standardized benchmarks, and productive integration of HOGNNs into broader graph learning ecosystems.

Abstract

Higher-order graph neural networks (HOGNNs) and the related architectures from Topological Deep Learning are an important class of GNN models that harness polyadic relations between vertices beyond plain edges. They have been used to eliminate issues such as over-smoothing or over-squashing, to significantly enhance the accuracy of GNN predictions, to improve the expressiveness of GNN architectures, and for numerous other goals. A plethora of HOGNN models have been introduced, and they come with diverse neural architectures, and even with different notions of what the "higher-order" means. This richness makes it very challenging to appropriately analyze and compare HOGNN models, and to decide in what scenario to use specific ones. To alleviate this, we first design an in-depth taxonomy and a blueprint for HOGNNs. This facilitates designing models that maximize performance. Then, we use our taxonomy to analyze and compare the available HOGNN models. The outcomes of our analysis are synthesized in a set of insights that help to select the most beneficial GNN model in a given scenario, and a comprehensive list of challenges and opportunities for further research into more powerful HOGNNs.

Figures (11)

• Figure 1: Computational structure of aggregations for a given vertex in simple 1-hop GNNs (it includes 1-hop neighbors and a given vertex). Graphs $G_1$ and $G_2$ are non-isomorphic but cannot be distinguished by a simple GCN.
• Figure 2: A blueprint of a higher-order graph data model (HOGDM).
• Figure 3: A taxonomy of higher-order graph data models (HOGDMs).
• Figure 4: A taxonomy of higher-order GNN architectures.
• Figure 5: A typical HOGNN data pipeline. (1) The input dataset is a set of one or more graphs $\{G=(V,E,\textbf{x})\}$. (2) The input is lifted to a selected HOGDM (HOGDMs are detailed in Section \ref{['chap:graph_data_models']}). Vectors in the red ovals indicate $\mathbb{R}^k$-valued features. In the HOGDM, we commonly have features for higher-order structures, for example, edges and hyperedges. (3) The constructed HOGNN architecture transforms features (HOGNNs are detailed in Section \ref{['chap:from_gdm_to_repL']}). (4) Final features are fed to downstream tasks, for example, node prediction.

Theorems & Definitions (23)

• Definition 4.1: Graph data model lifting
• Definition 5.1
• Definition 5.2
• Definition 5.3
• Example 5.4
• Proposition 5.5
• Definition 5.6: bodnar_weisfeiler_2022
• Definition 5.7
• Definition 5.8
• Proposition 5.9