Negative-Free Self-Supervised Gaussian Embedding of Graphs

TL;DR

This study proposes a negative-free objective to achieve uniformity, inspired by the fact that points distributed according to a normalized isotropic Gaussian are uniformly spread across the unit hypersphere, which can minimize the distance between the distribution of learned representations and the isotropic Gaussian distribution to promote the uniformity of node representations.

Abstract

Graph Contrastive Learning (GCL) has recently emerged as a promising graph self-supervised learning framework for learning discriminative node representations without labels. The widely adopted objective function of GCL benefits from two key properties: \emph{alignment} and \emph{uniformity}, which align representations of positive node pairs while uniformly distributing all representations on the hypersphere. The uniformity property plays a critical role in preventing representation collapse and is achieved by pushing apart augmented views of different nodes (negative pairs). As such, existing GCL methods inherently rely on increasing the quantity and quality of negative samples, resulting in heavy computational demands, memory overhead, and potential class collision issues. In this study, we propose a negative-free objective to achieve uniformity, inspired by the fact that points distributed according to a normalized isotropic Gaussian are uniformly spread across the unit hypersphere. Therefore, we can minimize the distance between the distribution of learned representations and the isotropic Gaussian distribution to promote the uniformity of node representations. Our method also distinguishes itself from other approaches by eliminating the need for a parameterized mutual information estimator, an additional projector, asymmetric structures, and, crucially, negative samples. Extensive experiments over seven graph benchmarks demonstrate that our proposal achieves competitive performance with fewer parameters, shorter training times, and lower memory consumption compared to existing GCL methods.

Figures (6)

• Figure 1: Overview of our proposed graph self-supervised learning framework SSGE. For a given attributed graph $\left(\boldsymbol{A}, \boldsymbol{X}\right)$, we first generate two distinct views $(\boldsymbol{\widetilde{A^1}}, \boldsymbol{\widetilde{X^1}})$, $(\boldsymbol{\widetilde{A^2}}, \boldsymbol{\widetilde{X^2}})$ through random augmentations $t^1$, $t^2$. These two views are subsequently fed into a shared GNN encoder $f_\theta$ to extract batch-normalized node representations $\boldsymbol{Z}^1$, $\boldsymbol{Z}^2$. The alignment loss $\mathcal{L}_{ali}$ and the uniformity loss $\mathcal{L}_{uni}$ are applied on $\boldsymbol{Z}^1$, $\boldsymbol{Z}^2$. Here, $\mathcal{L}_{uni}$ aims to minimize the 2-Wasserstein distance between the distribution of learned representations and the isotropic Gaussian distribution $\mathcal{N}\left( \boldsymbol{0}, \boldsymbol{I}_d\right)$.
• Figure 2: Visualization of the covariance matrix (absolute value) of learned representations on Cora.
• Figure 3: Visualizing the alignment and uniformity of node representations on Cora.
• Figure 4: Visualizing the alignment and uniformity of node representations on CiteSeer.
• Figure 5: Impact of the trade-off $\lambda$ and representation dimension.

Theorems & Definitions (4)

• Theorem 1: Hyperspherical Uniformity HU
• proof
• Lemma 1
• proof