Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LEDA: Latent Semantic Distribution Alignment for Multi-domain Graph Pre-training

Lianze Shan, Jitao Zhao, Dongxiao He, Siqi Liu, Jiaxu Cui, Weixiong Zhang

TL;DR

A novel Latent sEmantic Distribution Alignment (LEDA) model is proposed for universal graph pre-training, which introduces a dimension projection unit to adaptively align diverse domain features into a shared semantic space with minimal information loss and designs a variational semantic inference module to obtain the shared latent distribution.

Abstract

Recent advances in generic large models, such as GPT and DeepSeek, have motivated the introduction of universality to graph pre-training, aiming to learn rich and generalizable knowledge across diverse domains using graph representations to improve performance in various downstream applications. However, most existing methods face challenges in learning effective knowledge from generic graphs, primarily due to simplistic data alignment and limited training guidance. The issue of simplistic data alignment arises from the use of a straightforward unification for highly diverse graph data, which fails to align semantics and misleads pre-training models. The problem with limited training guidance lies in the arbitrary application of in-domain pre-training paradigms to cross-domain scenarios. While it is effective in enhancing discriminative representation in one data space, it struggles to capture effective knowledge from many graphs. To address these challenges, we propose a novel Latent sEmantic Distribution Alignment (LEDA) model for universal graph pre-training. Specifically, we first introduce a dimension projection unit to adaptively align diverse domain features into a shared semantic space with minimal information loss. Furthermore, we design a variational semantic inference module to obtain the shared latent distribution. The distribution is then adopted to guide the domain projection, aligning it with shared semantics across domains and ensuring cross-domain semantic learning. LEDA exhibits strong performance across a broad range of graphs and downstream tasks. Remarkably, in few-shot cross-domain settings, it significantly outperforms in-domain baselines and advanced universal pre-training models.

LEDA: Latent Semantic Distribution Alignment for Multi-domain Graph Pre-training

TL;DR

A novel Latent sEmantic Distribution Alignment (LEDA) model is proposed for universal graph pre-training, which introduces a dimension projection unit to adaptively align diverse domain features into a shared semantic space with minimal information loss and designs a variational semantic inference module to obtain the shared latent distribution.

Abstract

Recent advances in generic large models, such as GPT and DeepSeek, have motivated the introduction of universality to graph pre-training, aiming to learn rich and generalizable knowledge across diverse domains using graph representations to improve performance in various downstream applications. However, most existing methods face challenges in learning effective knowledge from generic graphs, primarily due to simplistic data alignment and limited training guidance. The issue of simplistic data alignment arises from the use of a straightforward unification for highly diverse graph data, which fails to align semantics and misleads pre-training models. The problem with limited training guidance lies in the arbitrary application of in-domain pre-training paradigms to cross-domain scenarios. While it is effective in enhancing discriminative representation in one data space, it struggles to capture effective knowledge from many graphs. To address these challenges, we propose a novel Latent sEmantic Distribution Alignment (LEDA) model for universal graph pre-training. Specifically, we first introduce a dimension projection unit to adaptively align diverse domain features into a shared semantic space with minimal information loss. Furthermore, we design a variational semantic inference module to obtain the shared latent distribution. The distribution is then adopted to guide the domain projection, aligning it with shared semantics across domains and ensuring cross-domain semantic learning. LEDA exhibits strong performance across a broad range of graphs and downstream tasks. Remarkably, in few-shot cross-domain settings, it significantly outperforms in-domain baselines and advanced universal pre-training models.
Paper Structure (17 sections, 1 theorem, 17 equations, 2 figures, 9 tables)

This paper contains 17 sections, 1 theorem, 17 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Preliminary
  4. Methodology
  5. Domain Projection Unit
  6. Analysis of graph pre-training
  7. Latent Distribution Alignment
  8. Experiments
  9. Experimental Settings
  10. Performance Analysis
  11. Model Analysis
  12. Conclusion
  13. Experimental Settings
  14. Implementation Details
  15. Supplementary Experimental Results
  16. ...and 2 more sections

Key Result

Proposition 1

Let the joint distribution over samples from two domains $\mathcal{D}_{\mathcal{G}^i}$ and $\mathcal{D}_{\mathcal{G}^j}$ be modeled as $p(x, x') = e^{s_{ij}} / Z$, where $s_{ij} = f(x, x')$ is a similarity score and $Z = \sum_{x, x'} e^{s_{ij}}$ (or $\int e^{s_{ij}} dx dx'$ in continuous case) is th where $\Delta = \mathbb{E}[\xi(x, x')]$.

Figures (2)

  • Figure 1: Overview of the proposed LEDA. Given the train dataset $\mathcal{T}_\mathcal{G}=\{\mathcal{G}^1, \mathcal{G}^2, \dots, \mathcal{G}^t\}$, we first get their initial projection basis set $\mathcal{P}_\mathcal{G}=\{\textbf{V}^1,\textbf{V}^2, \dots, \textbf{V}^t\}$ by SVD. Subsequently, this set of projection basis is processed by a learnable multi-layer perceptron, which is optimized by $\mathcal{L}_{\text{align}} = \mathcal{L}_{\text{recon}} + \lambda \cdot \mathcal{L}_{\text{ortho}}$. Furthermore, we encode the unified feature processed by DPU through a single-layer GCN and align the posterior distribution with a shared latent distribution. Finally, we jointly optimize the parameters of DPU and LDA using the loss function $\mathcal{L}_{\text{total}}$.
  • Figure 2: Performance on cross-domain few-shot node classification. The red line denotes our method. For traditional in-domain methods, we simply unify the input data dimensions by SVD.

Theorems & Definitions (2)

  • Definition 1
  • Proposition 1