Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Graph Quantized Tokenizers

Limei Wang, Kaveh Hassani, Si Zhang, Dongqi Fu, Baichuan Yuan, Weilin Cong, Zhigang Hua, Hao Wu, Ning Yao, Bo Long

TL;DR

Graphs require effective tokenization for Transformer-based models. GQT introduces a graph tokenizer that learns hierarchical discrete tokens via Residual Vector Quantization (RVQ) and multi-task self-supervised learning, decoupled from the Transformer. It uses semantic edges and Personalized PageRank (PPR)-driven sequences to expose long-range dependencies and token modulation to enrich representations. The approach yields a Transformer encoder that achieves state-of-the-art performance on 20 of 22 benchmarks, including large-scale and long-range graphs, while offering substantial memory reductions. This work advances scalable Graph Foundational Models and opens doors to integrating graph tokenizers with LLMs for unified graph-based AI systems.

Abstract

Transformers serve as the backbone architectures of Foundational Models, where domain-specific tokenizers allow them to adapt to various domains. Graph Transformers (GTs) have recently emerged as leading models in geometric deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. However, the development of tokenizers for graphs has lagged behind other modalities. To address this, we introduce GQT (\textbf{G}raph \textbf{Q}uantized \textbf{T}okenizer), which decouples tokenizer training from Transformer training by leveraging multi-task graph self-supervised learning, yielding robust and generalizable graph tokens. Furthermore, the GQT utilizes Residual Vector Quantization (RVQ) to learn hierarchical discrete tokens, resulting in significantly reduced memory requirements and improved generalization capabilities. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 20 out of 22 benchmarks, including large-scale homophilic and heterophilic datasets.

Learning Graph Quantized Tokenizers

TL;DR

Graphs require effective tokenization for Transformer-based models. GQT introduces a graph tokenizer that learns hierarchical discrete tokens via Residual Vector Quantization (RVQ) and multi-task self-supervised learning, decoupled from the Transformer. It uses semantic edges and Personalized PageRank (PPR)-driven sequences to expose long-range dependencies and token modulation to enrich representations. The approach yields a Transformer encoder that achieves state-of-the-art performance on 20 of 22 benchmarks, including large-scale and long-range graphs, while offering substantial memory reductions. This work advances scalable Graph Foundational Models and opens doors to integrating graph tokenizers with LLMs for unified graph-based AI systems.

Abstract

Transformers serve as the backbone architectures of Foundational Models, where domain-specific tokenizers allow them to adapt to various domains. Graph Transformers (GTs) have recently emerged as leading models in geometric deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. However, the development of tokenizers for graphs has lagged behind other modalities. To address this, we introduce GQT (\textbf{G}raph \textbf{Q}uantized \textbf{T}okenizer), which decouples tokenizer training from Transformer training by leveraging multi-task graph self-supervised learning, yielding robust and generalizable graph tokens. Furthermore, the GQT utilizes Residual Vector Quantization (RVQ) to learn hierarchical discrete tokens, resulting in significantly reduced memory requirements and improved generalization capabilities. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 20 out of 22 benchmarks, including large-scale homophilic and heterophilic datasets.

Paper Structure

This paper contains 22 sections, 17 equations, 1 figure, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Self-Supervised Graph Tokenization
  5. Tokenizer Properties
  6. Training
  7. Graph Transformer
  8. Graph Serialization
  9. Token Modulation
  10. Transformer Encoder
  11. Experiments
  12. Comparison with State-of-the-Art
  13. Ablation Study
  14. Conclusion
  15. Preliminaries
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Overview of our proposed Graph Quantization Transformer (GQT) consisting of three main components: (1) a GNN to encode local interactions, (2) vector quantization for compact representation, and (3) generative and contrastive heads for robust representation learning. We also utilize a Transformers encoder to model long-range interactions. We augment the graph with semantic edges (dashed lines) and generate a sequence for each node based on Personalized PageRank scores. We then modulate the tokens through hierarchical encoding and structural gating, and feed them into the Transformer and aggregate the learned representations through an attention module before passing it to the classification head.