HopFormer: Sparse Graph Transformers with Explicit Receptive Field Control

TL;DR

The paper addresses whether graph Transformers require explicit positional or structural encodings and dense global attention. It introduces HopFormer, a graph Transformer that uses edge-to-node augmentation and head-specific $n$-hop masks to inject topology while preserving the standard Transformer architecture. Theoretical results show that masked attention suffices to convey graph structure and that multi-hop heads enhance expressiveness; empirically, HopFormer achieves competitive or superior performance with linear-time sparsity and demonstrates stable results on graphs with strong small-world properties. This work offers a principled, efficient alternative to encoding-heavy or fully dense graph Transformers, with practical implications for scalable graph representation learning.

Abstract

Graph Transformers typically rely on explicit positional or structural encodings and dense global attention to incorporate graph topology. In this work, we show that neither is essential. We introduce HopFormer, a graph Transformer that injects structure exclusively through head-specific n-hop masked sparse attention, without the use of positional encodings or architectural modifications. This design provides explicit and interpretable control over receptive fields while enabling genuinely sparse attention whose computational cost scales linearly with mask sparsity. Through extensive experiments on both node-level and graph-level benchmarks, we demonstrate that our approach achieves competitive or superior performance across diverse graph structures. Our results further reveal that dense global attention is often unnecessary: on graphs with strong small-world properties, localized attention yields more stable and consistently high performance, while on graphs with weaker small-world effects, global attention offers diminishing returns. Together, these findings challenge prevailing assumptions in graph Transformer design and highlight sparsity-controlled attention as a principled and efficient alternative.

Figures (6)

• Figure 1: An illustration of the proposed edge-to-node augmentation, node–edge tokenization, and head-specific $n$-hop sparse attention mechanism.
• Figure 2: Small-world measures of (a) node-level and (b) graph-level benchmark datasets. Graph-level datasets generally exhibit weaker small-world characteristics than node-level datasets. The shaded region denotes the standard deviation of the linear fit.
• Figure 3: Rank trajectories of different methods across node-level datasets. Our method maintains consistently strong ranks, while other approaches exhibit larger variability as datasets' small-world characteristics change.
• Figure 4: Per-method rank distributions across low, mid, and high terciles of small-world measures: clustering coefficient $\bar{C}$ (top) and average shortest-path length $\bar{\ell}$ (bottom). Lower ranks indicate better performance. Boxes indicate the interquartile range, center lines denote the median, and whiskers show the data range (excluding outliers).
• Figure 5: Rank trajectories of different methods across graph-level datasets. In contrast to node-level benchmarks, the top-performing methods exhibit consistently similar ranks, indicating that when small-world effects are weaker, Transformer-based models tend to achieve comparable performance.

Theorems & Definitions (6)

• Theorem 1: Topology-Constrained Information Flow
• Corollary 1: Sufficiency of Mask-Based Structure Injection
• Theorem 2: Strict Expressiveness Gain of Multi-$n$-Hop Heads
• Corollary 2: Multi-Scale Representation in a Single Layer
• proof
• proof