ExGRG: Explicitly-Generated Relation Graph for Self-Supervised Representation Learning

TL;DR

This paper tackles graph self-supervised representation learning by addressing the limitations of augmentation-based invariance in graphs. It introduces ExGRG, a non-contrastive SSL framework that explicitly generates a compositional relation graph $\mathbf{G}$ by aggregating multiple sources (e.g., $\mathbf{G}^k, \mathbf{G}^A, \mathbf{G}^L, \mathbf{G}^R, \mathbf{G}^S, \mathbf{G}^O$) and optimizes an end-to-end loss $\mathcal{L}_{ETE}$ that includes $\mathcal{L}_V$, $\mathcal{L}_C$, $\mathcal{L}_{I'}$, $\mathcal{L}_O$, and $\mathcal{L}_R$. From a Laplacian Eigenmap perspective, explicit $\mathbf{G}$ mitigates rank deficiencies of augmentation-only graphs; from an EM view, the E-step (relation graph generation) and M-step (encoder update) are unified in a single gradient-based procedure. Extensive experiments on nine node-classification datasets show ExGRG consistently outperforms state-of-the-art SSL methods, validating the benefit of explicit relation graphs and multi-source priors for graph representation learning. This approach lowers reliance on potentially semantic-altering augmentations and scales with soft, learnable relational signals that capture higher-order graph structure and clustering information, improving robustness across diverse graph datasets.

Abstract

Self-supervised Learning (SSL) has emerged as a powerful technique in pre-training deep learning models without relying on expensive annotated labels, instead leveraging embedded signals in unlabeled data. While SSL has shown remarkable success in computer vision tasks through intuitive data augmentation, its application to graph-structured data poses challenges due to the semantic-altering and counter-intuitive nature of graph augmentations. Addressing this limitation, this paper introduces a novel non-contrastive SSL approach to Explicitly Generate a compositional Relation Graph (ExGRG) instead of relying solely on the conventional augmentation-based implicit relation graph. ExGRG offers a framework for incorporating prior domain knowledge and online extracted information into the SSL invariance objective, drawing inspiration from the Laplacian Eigenmap and Expectation-Maximization (EM). Employing an EM perspective on SSL, our E-step involves relation graph generation to identify candidates to guide the SSL invariance objective, and M-step updates the model parameters by integrating the derived relational information. Extensive experimentation on diverse node classification datasets demonstrates the superiority of our method over state-of-the-art techniques, affirming ExGRG as an effective adoption of SSL for graph representation learning.

Figures (14)

• Figure 1: The comprehensive architecture of ExGRG. We incorporate augmented views $\mathcal{G}^{(1)}$ and $\mathcal{G}^{(2)}$ from source graph $\mathcal{G}^{(s)}$ to compose an input graph $\mathcal{G}$, which is then fed into encoder $f_\theta$ and expander $g_\phi$ to produce representations ${\bm{H}}$ and embeddings ${\bm{Z}}$. We explicitly generate a compositional relation graph ${\bm{G}}$, aggregated from various intermediate relation graphs, to guide the invariance term $\mathcal{L}_{I'}$ instead of relying solely on augmentations.
• Figure 2: Accuracy vs. iterations over 20 trials throughout the SSL pre-training.
• Figure 3: Impact of altering $D_{H}$ on ranks and downstream accuracy for Amazon Computers.
• Figure 4: corr${\bm{H}}$ vs. iterations throughout the SSL pre-training.
• Figure 5: corr${\bm{Z}}$ vs. iterations throughout the SSL pre-training.