Scalable Deep Graph Clustering with Random-walk based Self-supervised Learning

TL;DR

This paper relates the Laplacian smoothing to the Generalized PageRank, and applies a random-walk based algorithm as a scalable graph filter, which forms the basis for the scalable deep clustering algorithm, RwSL.

Abstract

Web-based interactions can be frequently represented by an attributed graph, and node clustering in such graphs has received much attention lately. Multiple efforts have successfully applied Graph Convolutional Networks (GCN), though with some limits on accuracy as GCNs have been shown to suffer from over-smoothing issues. Though other methods (particularly those based on Laplacian Smoothing) have reported better accuracy, a fundamental limitation of all the work is a lack of scalability. This paper addresses this open problem by relating the Laplacian smoothing to the Generalized PageRank and applying a random-walk based algorithm as a scalable graph filter. This forms the basis for our scalable deep clustering algorithm, RwSL, where through a self-supervised mini-batch training mechanism, we simultaneously optimize a deep neural network for sample-cluster assignment distribution and an autoencoder for a clustering-oriented embedding. Using 6 real-world datasets and 6 clustering metrics, we show that RwSL achieved improved results over several recent baselines. Most notably, we show that RwSL, unlike all other deep clustering frameworks, can continue to scale beyond graphs with more than one million nodes, i.e., handle web-scale. We also demonstrate how RwSL could perform node clustering on a graph with 1.8 billion edges using only a single GPU.

Figures (6)

• Figure 1: RwSL framework architecture. $\Tilde{X}$ is the input filtered data; $\hat{X}$ is the reconstructed data from the decoder. $Z^{l}$ and $D^{l}$ are the latent representation learned from the encoder and decoder. $H^{l}$ is the latent representation from co-train self-supervised DNN module. $P_{Z}$ and $P_{H}$ are respectively the probability distribution of sample assignment to clusters from the last layer of encoder and co-train MLP. The target distribution $T$, initially calculated from $P_{Z}$, will guide the optimization of both auto-encoder and co-train DNN modules.
• Figure 2: (a) Laplacian eigenvalues of PPR filters $\lambda^{\Tilde{L}_{PPR}}_i=1 - \frac{\alpha}{1-(1-\alpha)(1-\lambda_{i})}$ with spectrum $\lambda_{i} \in [0, 2)$; (b) Sorted by index, Laplacian eigenvalues $\lambda^{\Tilde{L}_{sym}}_{i}$ of a GCN filter and Laplacian eigenvalues $\lambda^{\Tilde{L}_{PPR}}_i$ of PPR filters with multiple $\alpha$ on Cora; (c) Teleport probability $\alpha$ influence on clustering metrics for Cora
• Figure 3: PPR filter weights $w_{l}$ for different hops of propagation
• Figure 4: (a) Training Process on Coauthor CS; (b) Scalability of Different Frameworks: Training Time vs. Graph Size
• Figure 5: Cora t-SNE for different features/embeddings. Each color represents a distinct class.

Theorems & Definitions (4)

• Claim 1
• proof
• Claim 2
• proof