Dual-perspective Cross Contrastive Learning in Graph Transformers

TL;DR

DC-GCL tackles the scarcity and unreliability of positive samples in graph contrastive learning by introducing dual-perspective cross-graph contrastive learning with data- and model-perspective augmentations, controllable data augmentation, and pruning-based model augmentation, all built around a Graph Transformer encoder. It leverages a multi-view NT-Xent loss to fuse four correlated representations and three novel augmentation strategies to preserve semantics while increasing diversity. Comprehensive experiments on unsupervised and transfer learning tasks show state-of-the-art or competitive performance across diverse graphs, with ablations confirming the importance of each component. The work highlights the practical impact of transformer-based graph encoders and carefully designed augmentations for robust graph representation learning.

Abstract

Graph contrastive learning (GCL) is a popular method for leaning graph representations by maximizing the consistency of features across augmented views. Traditional GCL methods utilize single-perspective i.e. data or model-perspective) augmentation to generate positive samples, restraining the diversity of positive samples. In addition, these positive samples may be unreliable due to uncontrollable augmentation strategies that potentially alter the semantic information. To address these challenges, this paper proposed a innovative framework termed dual-perspective cross graph contrastive learning (DC-GCL), which incorporates three modifications designed to enhance positive sample diversity and reliability: 1) We propose dual-perspective augmentation strategy that provide the model with more diverse training data, enabling the model effective learning of feature consistency across different views. 2) From the data perspective, we slightly perturb the original graphs using controllable data augmentation, effectively preserving their semantic information. 3) From the model perspective, we enhance the encoder by utilizing more powerful graph transformers instead of graph neural networks. Based on the model's architecture, we propose three pruning-based strategies to slightly perturb the encoder, providing more reliable positive samples. These modifications collectively form the DC-GCL's foundation and provide more diverse and reliable training inputs, offering significant improvements over traditional GCL methods. Extensive experiments on various benchmarks demonstrate that DC-GCL consistently outperforms different baselines on various datasets and tasks.

Figures (5)

• Figure 1: Comparison of contrastive learning methods with proposed DC-GCL. Our DC-GCL introduces a comprehensive GT-based contrastive learning method which uses dual-perspective augmentation strategy. These novel designs significantly increase the generation of positive samples, with the aim of improving the model's ability to learn representations across diverse datasets.
• Figure 2: Overview the architecture of DC-GCL. We adopt data augmentations to obtain two correlated graphs, and send these graphs to the GT-based encoder and its perturbed version to get the positive samples. During this process, we propose three controllable data augmentation and prune-based model augmentation methods respectively. A comprehensive introduction for these methods is provided in Section \ref{['sec:data_aug']} and \ref{['sec:model_aug']}.
• Figure 3: $\mathcal{L}_{\text{ali}}$-$\mathcal{L}_{\text{uni}}$ visualization for GraphCL, SimGRACE, and DC-GCL for MUTAG and PROTEINS datasets. Numbers around the points in the figure represent the epochs. For both properties alignment and uniformity, the lower the better.
• Figure 4: Analysis of the Augmentation Strategy. We report unsupervised learning accuracy improvement(%) when contrasting different combination of data (vertical axis) and model (horizontal axis) augmentation. Deeper colors indicate better performance improves. "Identity" denotes without any augmentation methods for contrastive learning.
• Figure 5: Analysis of the Augmentation Ratio(%). Performance of various augmentation ratios. We choose the best augmentation strategy each dataset for analysis.