Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers

Jinsong Chen, Hanpeng Liu, John E. Hopcroft, Kun He

TL;DR

Extensive experimental results demonstrate the superiority of GCFormer in node classification, when compared to representative graph neural networks (GNNs) and graph Transformers.

Abstract

While tokenized graph Transformers have demonstrated strong performance in node classification tasks, their reliance on a limited subset of nodes with high similarity scores for constructing token sequences overlooks valuable information from other nodes, hindering their ability to fully harness graph information for learning optimal node representations. To address this limitation, we propose a novel graph Transformer called GCFormer. Unlike previous approaches, GCFormer develops a hybrid token generator to create two types of token sequences, positive and negative, to capture diverse graph information. And a tailored Transformer-based backbone is adopted to learn meaningful node representations from these generated token sequences. Additionally, GCFormer introduces contrastive learning to extract valuable information from both positive and negative token sequences, enhancing the quality of learned node representations. Extensive experimental results across various datasets, including homophily and heterophily graphs, demonstrate the superiority of GCFormer in node classification, when compared to representative graph neural networks (GNNs) and graph Transformers.

Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers

TL;DR

Extensive experimental results demonstrate the superiority of GCFormer in node classification, when compared to representative graph neural networks (GNNs) and graph Transformers.

Abstract

While tokenized graph Transformers have demonstrated strong performance in node classification tasks, their reliance on a limited subset of nodes with high similarity scores for constructing token sequences overlooks valuable information from other nodes, hindering their ability to fully harness graph information for learning optimal node representations. To address this limitation, we propose a novel graph Transformer called GCFormer. Unlike previous approaches, GCFormer develops a hybrid token generator to create two types of token sequences, positive and negative, to capture diverse graph information. And a tailored Transformer-based backbone is adopted to learn meaningful node representations from these generated token sequences. Additionally, GCFormer introduces contrastive learning to extract valuable information from both positive and negative token sequences, enhancing the quality of learned node representations. Extensive experimental results across various datasets, including homophily and heterophily graphs, demonstrate the superiority of GCFormer in node classification, when compared to representative graph neural networks (GNNs) and graph Transformers.
Paper Structure (22 sections, 15 equations, 4 figures, 4 tables)

This paper contains 22 sections, 15 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Transformer
  4. Contrastive Learning on Graphs
  5. Preliminaries
  6. Node Classification
  7. Transformer
  8. Method
  9. Hybrid Token Generator
  10. Transformer-based Backbone
  11. Integrating Contrastive Learning
  12. Experiments
  13. Experimental Setup
  14. Performance Comparison
  15. Parameter Sensitivity Analysis
  16. ...and 7 more sections

Figures (4)

  • Figure 1: A toy example to illustrate the difference of the token generator between the token generator in our method and that used in the previous node tokenized graph Transformers. Previous methods only sample nodes with high similarity to construct token sequences. In contrast, our method introduces both positive and negative token sampling to preserve information carried by diverse nodes in the graph.
  • Figure 2: Performance of GCFormer with different sampling sizes on all datasets.
  • Figure 3: Performance of GCFormer with different $\alpha$ on all datasets.
  • Figure 4: Performances of GCFormer and its variants.