Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Best of Both Worlds: Advantages of Hybrid Graph Sequence Models

Ali Behrouz, Ali Parviz, Mahdi Karami, Clayton Sanford, Bryan Perozzi, Vahab Mirrokni

TL;DR

Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, and GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences.

Abstract

Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational power, and/or ability to capture long-range dependencies. Adopting these sequence models for graph-structured data has recently gained popularity as the alternative to Message Passing Neural Networks (MPNNs). There is, however, a lack of a common foundation about what constitutes a good graph sequence model, and a mathematical description of the benefits and deficiencies in adopting different sequence models for learning on graphs. To this end, we first present Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, consisting of three main steps: (1) Tokenization, which translates the graph into a set of sequences; (2) Local Encoding, which encodes local neighborhoods around each node; and (3) Global Encoding, which employs a scalable sequence model to capture long-range dependencies within the sequences. This framework allows us to understand, evaluate, and compare the power of different sequence model backbones in graph tasks. Our theoretical evaluations of the representation power of Transformers and modern recurrent models through the lens of global and local graph tasks show that there are both negative and positive sides for both types of models. Building on this observation, we present GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences. Our theoretical and experimental results support the design of GSM++, showing that GSM++ outperforms baselines in most benchmark evaluations.

Best of Both Worlds: Advantages of Hybrid Graph Sequence Models

TL;DR

Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, and GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences.

Abstract

Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational power, and/or ability to capture long-range dependencies. Adopting these sequence models for graph-structured data has recently gained popularity as the alternative to Message Passing Neural Networks (MPNNs). There is, however, a lack of a common foundation about what constitutes a good graph sequence model, and a mathematical description of the benefits and deficiencies in adopting different sequence models for learning on graphs. To this end, we first present Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, consisting of three main steps: (1) Tokenization, which translates the graph into a set of sequences; (2) Local Encoding, which encodes local neighborhoods around each node; and (3) Global Encoding, which employs a scalable sequence model to capture long-range dependencies within the sequences. This framework allows us to understand, evaluate, and compare the power of different sequence model backbones in graph tasks. Our theoretical evaluations of the representation power of Transformers and modern recurrent models through the lens of global and local graph tasks show that there are both negative and positive sides for both types of models. Building on this observation, we present GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences. Our theoretical and experimental results support the design of GSM++, showing that GSM++ outperforms baselines in most benchmark evaluations.

Paper Structure

This paper contains 50 sections, 24 theorems, 25 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Encoding Graphs to Sequences: A Unified Model
  3. Tokenization
  4. Local Encoding
  5. Global Encoding
  6. Choosing a Sequence Model
  7. Counting Tasks on Graphs
  8. Importance of Node Ordering
  9. Connectivity Tasks on Graphs
  10. Choosing the Right Tokenizer: Node, Edge, or Subgraph
  11. Enhancing Graph to Sequence Models
  12. Hierarchical Affinity Clustering (HAC) for Tokenization
  13. Hybrid Models
  14. Mixture of Tokenization (MoT)
  15. GSM++: A Powerful Hybrid Model
  16. ...and 35 more sections

Key Result

Proposition 1

A Transformer model based on non-causal attention and without proper positional encodings is immediately unable to count.

Figures (4)

  • Figure 1: Overview of Graph Sequence Model (GSM). GSM Consists of three stages: (1) Tokenization, (2) Local Encoding, and (3) Global Encoding. We provide a foundation for strengths and weaknesses of different tokenizations and sequence models. Finally, we present three methods to enhance the power of GSMs.
  • Figure 2: Overview of GSM++. GSM++ is a special instance of GSMs that uses: (1) HAC tokenization, (2) hierarchical PE, and (3) a hybrid sequence model.
  • Figure 3: Normalized score of different combination of tokenization and global encoder (sequence models). Even TTT + HAC is in Top-3 only in 3/7 datasets.
  • Figure 4: The effect of number of nodes on the preprocessing time for the construction of positional encodings.

Theorems & Definitions (38)

  • Proposition 1: Proposition 6.1 of barbero2024transformers
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Corollary 2
  • Corollary 3: Corollary 3.3 of sht24
  • Theorem 3
  • Definition 1
  • Theorem 4
  • Theorem 5
  • ...and 28 more