MUG: Meta-path-aware Universal Heterogeneous Graph Pre-Training

TL;DR

A novel Meta-path-aware Universal heterogeneous Graph pre-training approach that trains a shared encoder to capture consistent structural patterns across diverse meta-path views rather than relying on dataset-specific aggregation strategies, while a global objective encourages discriminability and reduces dataset-specific biases.

Abstract

Universal graph pre-training has emerged as a key paradigm in graph representation learning, offering a promising way to train encoders to learn transferable representations from unlabeled graphs and to effectively generalize across a wide range of downstream tasks. However, recent explorations in universal graph pre-training primarily focus on homogeneous graphs and it remains unexplored for heterogeneous graphs, which exhibit greater structural and semantic complexity. This heterogeneity makes it highly challenging to train a universal encoder for diverse heterogeneous graphs: (i) the diverse types with dataset-specific semantics hinder the construction of a unified representation space; (ii) the number and semantics of meta-paths vary across datasets, making encoding and aggregation patterns learned from one dataset difficult to apply to others. To address these challenges, we propose a novel Meta-path-aware Universal heterogeneous Graph pre-training (MUG) approach. Specifically, for challenge (i), MUG introduces a input unification module that integrates information from multiple node and relation types within each heterogeneous graph into a unified representation.This representation is then projected into a shared space by a dimension-aware encoder, enabling alignment across graphs with diverse schemas.Furthermore, for challenge (ii), MUG trains a shared encoder to capture consistent structural patterns across diverse meta-path views rather than relying on dataset-specific aggregation strategies, while a global objective encourages discriminability and reduces dataset-specific biases. Extensive experiments demonstrate the effectiveness of MUG on some real datasets.

Figures (3)

• Figure 1: Homophily ratios on six datasets. Cora and CiteSeer Cora are homogeneous graphs, while ACM ACM, DBLP magnn, AMiner Aminer and Freebase freebase are heterogeneous. For heterogeneous graphs, we compute the homophily ratio for each meta-path-based adjacency matrix and report the average.
• Figure 2: Overview of the MUG. Given a heterogeneous graph with diverse node types, relation types, and attributes, we first get embedding $\mathbf{Z}^\text{struct}$ by contextual structural encoding. The embedding is concatenated with the original node attributes $\mathbf{X}$ to obtain unified representation $\tilde{\mathbf{X}}$, which contains diverse type and attribute information. Then, a dimension-aware encoder is applied to align representation spaces and produce the unified input $\mathbf{X}^\text{unify}$. Finally, a shared GNN encoder is used to encode each masked adjacency matrix $\tilde{\mathbf{A}}^\phi$ with the unified input $\mathbf{X}^\text{unify}$, producing $\mathbf{Z}^\phi$. The model is optimized by three objectives: a dimension alignment loss $\mathcal{L}_\text{align}$, a meta-path masked reconstruction loss $\mathcal{L}^\phi$ and a global scattering regularization loss $\mathcal{L}_\text{scatter}$.
• Figure 3: Ablation results for cross-domain node classification, trained on Freebase and evaluated on four datasets.