HHGT: Hierarchical Heterogeneous Graph Transformer for Heterogeneous Graph Representation Learning

TL;DR

HHGT addresses heterogeneous information networks by modeling distance- and type-based heterogeneity with a novel $(k,t)$-ring neighborhood and a hierarchical transformer architecture. The Ring2Token component constructs distance- and type-aware tokens, while the TRGT module stacks a Type-level Transformer and a Ring-level Transformer to produce robust node embeddings. Empirical results on ACM and MAG show state-of-the-art performance for node classification and clustering, with ablations confirming the value of distance- and type-aware components. The work provides a scalable, principled framework for heterogeneous graph representation learning with transformer-based architectures.

Abstract

Despite the success of Heterogeneous Graph Neural Networks (HGNNs) in modeling real-world Heterogeneous Information Networks (HINs), challenges such as expressiveness limitations and over-smoothing have prompted researchers to explore Graph Transformers (GTs) for enhanced HIN representation learning. However, research on GT in HINs remains limited, with two key shortcomings in existing work: (1) A node's neighbors at different distances in HINs convey diverse semantics. Unfortunately, existing methods ignore such differences and uniformly treat neighbors within a given distance in a coarse manner, which results in semantic confusion. (2) Nodes in HINs have various types, each with unique semantics. Nevertheless, existing methods mix nodes of different types during neighbor aggregation, hindering the capture of proper correlations between nodes of diverse types. To bridge these gaps, we design an innovative structure named (k,t)-ring neighborhood, where nodes are initially organized by their distance, forming different non-overlapping k-ring neighborhoods for each distance. Within each k-ring structure, nodes are further categorized into different groups according to their types, thus emphasizing the heterogeneity of both distances and types in HINs naturally. Based on this structure, we propose a novel Hierarchical Heterogeneous Graph Transformer (HHGT) model, which seamlessly integrates a Type-level Transformer for aggregating nodes of different types within each k-ring neighborhood, followed by a Ring-level Transformer for aggregating different k-ring neighborhoods in a hierarchical manner. Extensive experiments are conducted on downstream tasks to verify HHGT's superiority over 14 baselines, with a notable improvement of up to 24.75% in NMI and 29.25% in ARI for node clustering task on the ACM dataset compared to the best baseline.

Figures (11)

• Figure 1: The difference between existing HGT-based methods and our HHGT model: (a) A toy HIN; (b) Learning node representation through existing HGT-based methods; (c) Learning node representation through our HHGT model. Here, node $P_1$'s$0$-ring neighborhood, $1$-ring neighborhood, $2$-ring neighborhood are visually highlighted with yellow, blue, green colors, respectively. Within $1$-ring neighborhood, nodes of different types are separated by dashed lines.
• Figure 2: (a) Illustration of the framework for node classification task. (b) Diagram of the TRGT module incorporating both Ring-level Transformer and Type-level Transformer.
• Figure 3: Illustration of neighborhood partition by the Ring2Token module.
• Figure 4: Embedding visualization on ACM dataset, where our HHGT model shows much higher intra-class similarity.
• Figure 5: Embedding visualization on MAG dataset, where our HHGT model clearly separates papers from different published venues with well-defined boundaries.