Advancing Graph Neural Networks with HL-HGAT: A Hodge-Laplacian and Attention Mechanism Approach for Heterogeneous Graph-Structured Data

TL;DR

HL-HGAT advances graph neural networks by treating graphs as simplicial complexes and learning heterogeneous signals on $k$-simplices through HL-filters, simplicial projection, and simplicial attention pooling. The approach leverages a Hodge-Laplacian spectral framework with a polynomial HL-filter approximation to ensure locality, and introduces cross-dimensional interactions via simplicial projections and MSI. Empirical results across NP-hard tasks, graph classification, molecular regression, and neuroimaging demonstrate superior performance and interpretable SAP attention maps, while downsampling improves scalability. The work provides a flexible, multi-dimensional GNN architecture with strong applicability to biology, chemistry, vision, and neuroscience, along with insights into topological relevance via attention maps.

Abstract

Graph neural networks (GNNs) have proven effective in capturing relationships among nodes in a graph. This study introduces a novel perspective by considering a graph as a simplicial complex, encompassing nodes, edges, triangles, and $k$-simplices, enabling the definition of graph-structured data on any $k$-simplices. Our contribution is the Hodge-Laplacian heterogeneous graph attention network (HL-HGAT), designed to learn heterogeneous signal representations across $k$-simplices. The HL-HGAT incorporates three key components: HL convolutional filters (HL-filters), simplicial projection (SP), and simplicial attention pooling (SAP) operators, applied to $k$-simplices. HL-filters leverage the unique topology of $k$-simplices encoded by the Hodge-Laplacian (HL) operator, operating within the spectral domain of the $k$-th HL operator. To address computation challenges, we introduce a polynomial approximation for HL-filters, exhibiting spatial localization properties. Additionally, we propose a pooling operator to coarsen $k$-simplices, combining features through simplicial attention mechanisms of self-attention and cross-attention via transformers and SP operators, capturing topological interconnections across multiple dimensions of simplices. The HL-HGAT is comprehensively evaluated across diverse graph applications, including NP-hard problems, graph multi-label and classification challenges, and graph regression tasks in logistics, computer vision, biology, chemistry, and neuroscience. The results demonstrate the model's efficacy and versatility in handling a wide range of graph-based scenarios.

Figures (9)

• Figure 1: Illustration of simplices, boundary operators, and simplex downsampling. Panel (A) illustrates a graph with the $0$-, $1$-, $2$-simplices, while Panel (B) displays its corresponding $1$-st and $2$-nd boundary operators. Panel (C) demonstrates an example of the simplex downsampling and the update of the corresponding boundary operators. In the simplex assignment matrix, each row corresponds to the $k$-simplex of the downsampled graph, and each column corresponds to the $k$-simplex of the original graph.
• Figure 2: HL-filters and simplicial projection operators. (A--B) The series of panels, from left to right, display a pulse signal associated with either a node or an edge, followed by the corresponding signals filtered through HL-filters employing approximations based on the 1st, 2nd, and 3rd-order Laguerre polynomials. (C--D) The left and right panels respectively depict a pulse signal defined on either a node or an edge and their resulting signals after projection through a simplicial projection operator.
• Figure 3: HL-HGAT architecture. (A) Schematic representation of the Hodge-Laplacian Heterogeneous Graph Attention Network (HL-HGAT) architecture showcasing three key innovations: HL-filters, multi-simplicial interaction (MSI), and simplicial attention pooling (SAP). In each processing block, we initiate the workflow by applying HL-filters to signals from the $k_1$- and $k_2$-simplices from the preceding block. Subsequently, an MSI layer is employed to capture signal interactions between the $k_1$- and $k_2$-simplices. Following this, we implement an SAP layer, which involves updating the boundary operator and feature consolidation based on simplex attention. Finally, an output layer is designed for prediction. (B) Flow chart outlining the proposed simplex downsampling algorithm in Section \ref{['appx:msi']}. We employ the Graclus clustering algorithm dhillon2007weighted to derive the node assignment matrix. This is followed by an iterative three-step process (depicted in Fig. \ref{['fig:simplex']}(C)): 1) Initialization of the updated boundary operator for the $(k+1)$-th iteration; 2) Removal of non-existent $(k+1)$-simplices that contain nodes within the same node clusters and duplicated $(k+1)$-simplices; 3) Computation of the $(k+1)$-simplex assignment matrix using the updated boundary operator. (C) Schematic diagram illustrating the architecture of the Simplicial Attention Pooling (SAP). Within this framework, we compute self-attention and cross-attention for each simplex. The simplex signals are then modulated by attention mechanisms and pooled based on the assignment matrices.
• Figure 4: Traveling Salesman Problem (TSP). Node and edge features at every two HL-filter layers are visualized. The ground truth is positioned at the center of the figure. Node features are represented in grey, while edge features are highlighted in red.
• Figure 5: Feature space. The feature space learned from each GNN method is projected into a two-dimensional space using t-distributed stochastic neighbor embedding (t-SNE) for visualization purposes. (A) In the feature space of the Traveling Salesman Problem, red points indicate edges that do not belong to the shortest path, while cyan points represent edges that are part of the shortest path. (B) The feature space of the CIFAR10 dataset visualizes the 10 classes of natural images. (C) In the feature space of the Peptide-func dataset, the colored points indicate peptides with corresponding functions, while grey points indicate peptides that do not have these functions. (D) In the feature space of the ZINC dataset, the points represent molecules colored by the corresponding constrained solubility. Additionally, the bar plot illustrates the average and standard deviation of synthetic accessibility across all molecules in the three clusters. (E--F) The feature spaces of brain images are color-coded by brain age and general intelligence. Each point in these plots represents an individual subject. Each column corresponds to the feature space obtained from various GNN models, including GCN kipf2016semi, GAT velivckovic2017graph, GatedGCN bresson2017residual, GPSrampavsek2022recipe, dGCN zhao2022dynamic, BrainGNN li2021braingnn, and Hypergraph NN jo2021edge.