Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification

Jinsong Chen, Siyu Jiang, Kun He

TL;DR

A new graph Transformer called NTFormer is proposed, which introduces a novel token generator called Node2Par, which constructs various token sequences using different token elements for each node, eliminating the need for graph-specific modifications.

Abstract

Recently, the emerging graph Transformers have made significant advancements for node classification on graphs. In most graph Transformers, a crucial step involves transforming the input graph into token sequences as the model input, enabling Transformer to effectively learn the node representations. However, we observe that existing methods only express partial graph information of nodes through single-type token generation. Consequently, they require tailored strategies to encode additional graph-specific features into the Transformer to ensure the quality of node representation learning, limiting the model flexibility to handle diverse graphs. To this end, we propose a new graph Transformer called NTFormer to address this issue. NTFormer introduces a novel token generator called Node2Par, which constructs various token sequences using different token elements for each node. This flexibility allows Node2Par to generate valuable token sequences from different perspectives, ensuring comprehensive expression of rich graph features. Benefiting from the merits of Node2Par, NTFormer only leverages a Transformer-based backbone without graph-specific modifications to learn node representations, eliminating the need for graph-specific modifications. Extensive experiments conducted on various benchmark datasets containing homophily and heterophily graphs with different scales demonstrate the superiority of NTFormer over representative graph Transformers and graph neural networks for node classification.

NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification

TL;DR

A new graph Transformer called NTFormer is proposed, which introduces a novel token generator called Node2Par, which constructs various token sequences using different token elements for each node, eliminating the need for graph-specific modifications.

Abstract

Recently, the emerging graph Transformers have made significant advancements for node classification on graphs. In most graph Transformers, a crucial step involves transforming the input graph into token sequences as the model input, enabling Transformer to effectively learn the node representations. However, we observe that existing methods only express partial graph information of nodes through single-type token generation. Consequently, they require tailored strategies to encode additional graph-specific features into the Transformer to ensure the quality of node representation learning, limiting the model flexibility to handle diverse graphs. To this end, we propose a new graph Transformer called NTFormer to address this issue. NTFormer introduces a novel token generator called Node2Par, which constructs various token sequences using different token elements for each node. This flexibility allows Node2Par to generate valuable token sequences from different perspectives, ensuring comprehensive expression of rich graph features. Benefiting from the merits of Node2Par, NTFormer only leverages a Transformer-based backbone without graph-specific modifications to learn node representations, eliminating the need for graph-specific modifications. Extensive experiments conducted on various benchmark datasets containing homophily and heterophily graphs with different scales demonstrate the superiority of NTFormer over representative graph Transformers and graph neural networks for node classification.
Paper Structure (23 sections, 16 equations, 7 figures, 3 tables)

This paper contains 23 sections, 16 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Neural Network
  4. Graph Transformer
  5. Preliminaries
  6. Notations
  7. Graph Convolutional Network
  8. Transformer
  9. The Proposed Method
  10. Neighborhood-based Token Generator
  11. Node-based Token Generator
  12. Transformer-based Backbone
  13. Discussions on NTFormer
  14. Experiments
  15. Datasets
  16. ...and 8 more sections

Figures (7)

  • Figure 1: The overall framework of NTFormer. Specifically, NTFormer adopts the token sequences from different levels and views generated by Node2Par as the input. Then, NTFormer leverages a Transformer-based backbone with standard Transformer layers and the adaptive feature fusion module to learn final node representations from the constructed sequences.
  • Figure 2: A toy example for illustrating the difference of neighborhood tokens generated from topology and attribute views. Each node is associated with a three-dimensional feature vector. We can clearly observe the difference between the two neighborhood tokens constructed from different views.
  • Figure 3: Performances of NTFormer and its variants on all datasets.
  • Figure 4: Visualization of the average weight proportion for each sequence on different graphs.
  • Figure 5: Performances of NTFormer with different feature fusion strategies. MEA. denotes the average fusion. CON. denotes the direct concatenation. ADP. denotes the adaptive fusion.
  • ...and 2 more figures