Self-Attention Empowered Graph Convolutional Network for Structure Learning and Node Embedding

TL;DR

The paper tackles the challenge of learning robust node representations on graphs with varying levels of homophily by introducing GCN-SA, a framework that jointly learns a re-connected adjacency via self-attention and fuses node features with a modified transformer block. The re-connected graph ${\mathbf A}_*$ captures long-range dependencies by selecting reliable neighbors, while the modified transformer block enables effective multi-scale feature fusion without severe overfitting. Key contributions include (i) a self-attention–driven structure-learning module for edges, (ii) a tailored transformer block for node features, and (iii) a concatenation-based fusion strategy that leverages ego-, neighbor-, and reconnected-neighbor-embeddings. Empirical results on eight datasets show strong performance across heterophilic and homophilic graphs, along with favorable time complexity relative to iterative structure-learning baselines, signaling practical impact for scalable, versatile graph representation learning.

Abstract

In representation learning on graph-structured data, many popular graph neural networks (GNNs) fail to capture long-range dependencies, leading to performance degradation. Furthermore, this weakness is magnified when the concerned graph is characterized by heterophily (low homophily). To solve this issue, this paper proposes a novel graph learning framework called the graph convolutional network with self-attention (GCN-SA). The proposed scheme exhibits an exceptional generalization capability in node-level representation learning. The proposed GCN-SA contains two enhancements corresponding to edges and node features. For edges, we utilize a self-attention mechanism to design a stable and effective graph-structure-learning module that can capture the internal correlation between any pair of nodes. This graph-structure-learning module can identify reliable neighbors for each node from the entire graph. Regarding the node features, we modify the transformer block to make it more applicable to enable GCN to fuse valuable information from the entire graph. These two enhancements work in distinct ways to help our GCN-SA capture long-range dependencies, enabling it to perform representation learning on graphs with varying levels of homophily. The experimental results on benchmark datasets demonstrate the effectiveness of the proposed GCN-SA. Compared to other outstanding GNN counterparts, the proposed GCN-SA is competitive.

Figures (6)

• Figure 1: GCN-SA consists of three stages: (S1) re-connected graph ${\bf A}_{*}$ learning, (S2) fusional feature learning, and (S3) graph convolutional network with self-attention (GCN-SA). In (S1), we construct a re-connected adjacency matrix ${\bf A}_{*}$ through attention score learning. This process allows the reconnected graph to be gradually optimized with the evolution of node embeddings. In (S2), we design and employ a modified transformer block to perform feature vector fusion. In (S3), we combine the original and fusional feature vectors as the ego-embeddings ${\bf H}$. Then we perform feature aggregation on ${\bf H}$ using ${\bf A}_{*}$ and ${\bf A}$, respectively. Subsequently, the ${\bf H}$ and the results of feature aggregation are concatenated as the ${\bf H}^{cb}$, and we use a learnable weighted vector ${\bf w}$ to highlight the crucial dimensions of ${\bf H}^{cb}$. Finally, we reuse the modified transformer block to perform node embedding fusion.
• Figure 2: Flowchart of the re-connected adjacency matrix learning. The graph-structured data is a real-world graph, namely the Texas network, where color indicates the labels of nodes. We obtain a re-connected graph through attention score computing, sorting, and screening. Subsequently, the re-connected graph is optimized together with the evolution of node embeddings. Finally, we obtain a re-connected graph with high homophily
• Figure 3: Node classification accuracies (%) of the proposed GCN-SA versus the hyper-parameters K, r and $\epsilon$ on seven datasets.
• Figure 4: Comparisons of homophily ratio $h$ for graphs, including the original, initialized, and re-connected graphs.
• Figure 5: Visualization of the graph structures. The graph-structured data is a real-world graph, namely the Wisconsin network, where color indicates the labels of nodes. The original graph is shown in (a), and the re-connected graph learned by GCN-SA is shown in (c). We fade the remaining nodes and edges to emphasize the neighbors of the selected node in (b) and (d).