Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning

Jianqing Liang, Xinkai Wei, Min Chen, Zhiqiang Wang, Jiye Liang

TL;DR

The paper tackles the sensitivity of graph contrastive learning to random augmentations and the limitations of conventional GNN encoders. It introduces GTCA, a GNN-Transformer cooperative architecture that combines a GCN encoder with NodeFormer and a topology-structure view to produce augmentation-free, trustworthy representations via a multi-positive contrastive loss with temperature $\tau$. Theoretical analysis supports the trustworthiness of GTCA, and experiments on five benchmarks show state-of-the-art node classification performance while reducing reliance on perturbing graph augmentations. This approach offers scalable, high-fidelity graph representations for downstream tasks by integrating topology and cross-view information through dual encoders and a novel sampling-laden loss.

Abstract

Graph contrastive learning (GCL) has become a hot topic in the field of graph representation learning. In contrast to traditional supervised learning relying on a large number of labels, GCL exploits augmentation strategies to generate multiple views and positive/negative pairs, both of which greatly influence the performance. Unfortunately, commonly used random augmentations may disturb the underlying semantics of graphs. Moreover, traditional GNNs, a type of widely employed encoders in GCL, are inevitably confronted with over-smoothing and over-squashing problems. To address these issues, we propose GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning (GTCA), which inherits the advantages of both GNN and Transformer, incorporating graph topology to obtain comprehensive graph representations. Theoretical analysis verifies the trustworthiness of the proposed method. Extensive experiments on benchmark datasets demonstrate state-of-the-art empirical performance.

GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning

TL;DR

The paper tackles the sensitivity of graph contrastive learning to random augmentations and the limitations of conventional GNN encoders. It introduces GTCA, a GNN-Transformer cooperative architecture that combines a GCN encoder with NodeFormer and a topology-structure view to produce augmentation-free, trustworthy representations via a multi-positive contrastive loss with temperature . Theoretical analysis supports the trustworthiness of GTCA, and experiments on five benchmarks show state-of-the-art node classification performance while reducing reliance on perturbing graph augmentations. This approach offers scalable, high-fidelity graph representations for downstream tasks by integrating topology and cross-view information through dual encoders and a novel sampling-laden loss.

Abstract

Graph contrastive learning (GCL) has become a hot topic in the field of graph representation learning. In contrast to traditional supervised learning relying on a large number of labels, GCL exploits augmentation strategies to generate multiple views and positive/negative pairs, both of which greatly influence the performance. Unfortunately, commonly used random augmentations may disturb the underlying semantics of graphs. Moreover, traditional GNNs, a type of widely employed encoders in GCL, are inevitably confronted with over-smoothing and over-squashing problems. To address these issues, we propose GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning (GTCA), which inherits the advantages of both GNN and Transformer, incorporating graph topology to obtain comprehensive graph representations. Theoretical analysis verifies the trustworthiness of the proposed method. Extensive experiments on benchmark datasets demonstrate state-of-the-art empirical performance.

Paper Structure

This paper contains 18 sections, 2 theorems, 12 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. Graph Encoders
  6. Node Sampling
  7. Contrastive Loss Function
  8. Experiments
  9. Datasets
  10. Baselines
  11. Experimental Settings
  12. Node Classification Results
  13. Ablation Study
  14. Graph Encoders Analysis
  15. Sensitivity Analysis
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ be a graph dataset of $N$ nodes drawn i.i.d. according to an unknown distribution $\mathcal{D} .$ Let $\mathcal{M},\mathcal{N}$ be two distinct subsets of $[K]$, where $K$ is the number of multiple views. Assume empirical risk minimizers $(\hat{h}_{\mat where

Figures (6)

  • Figure 1: Augmentations on image (a) keep the semantic information, while augmentations on graphs (b) change the underlying semantic information.
  • Figure 2: The architecture of GTCA. Given a graph, the GNN encoder $f_\theta$ and Transformer encoder $g_\varphi$ generate node embeddings $\boldsymbol{H}_{\theta}$ and $\boldsymbol{H}_{\varphi}$. We apply two node embeddings to obtain $k$-NNs of $v_i$, i.e., $\mathcal{B}_i^{\theta}$, $\mathcal{B}_i^{\varphi}$, intersects which with the topological $k$-NNs $\mathcal{T}_i$ to obtain positive and negative pairs for node $v_i$ separately, i.e., $\mathcal{P}_i$ and $\mathcal{N}_i$. Finally, we employ the contrastive loss to achieve that the anchor nodes close to positive pairs and far from negative pairs.
  • Figure 3: The red nodes denote the positive pair and the blue dotted lines with arrows are negative pairs of the anchor.
  • Figure 4: Correct ratio of positive pairs with various $k$ values on Cora and Amazon-Photo datasets.
  • Figure 5: Accuracy vs. hyperparameters $k$, $E$ and $\lambda$ on Cora and Amazon-Photo datasets.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Proposition 1
  • proof