DeepWalk: Online Learning of Social Representations

TL;DR

DeepWalk introduces an online, scalable method for learning low-dimensional latent representations of graph vertices by treating short random walks as sentences and applying a SkipGram-like objective with hierarchical softmax. The resulting embeddings capture neighborhood structure and community membership, enabling strong Multi-label classification even with sparse labels and on web-scale graphs. The approach outperforms several baselines that rely on global graph views and is naturally parallelizable, making it practical for real-world networks. By bridging graph representation learning and language modeling, it opens avenues for improved modeling of networks and insights into language-based learning on graphs.

Abstract

We present DeepWalk, a novel approach for learning latent representations of vertices in a network. These latent representations encode social relations in a continuous vector space, which is easily exploited by statistical models. DeepWalk generalizes recent advancements in language modeling and unsupervised feature learning (or deep learning) from sequences of words to graphs. DeepWalk uses local information obtained from truncated random walks to learn latent representations by treating walks as the equivalent of sentences. We demonstrate DeepWalk's latent representations on several multi-label network classification tasks for social networks such as BlogCatalog, Flickr, and YouTube. Our results show that DeepWalk outperforms challenging baselines which are allowed a global view of the network, especially in the presence of missing information. DeepWalk's representations can provide $F_1$ scores up to 10% higher than competing methods when labeled data is sparse. In some experiments, DeepWalk's representations are able to outperform all baseline methods while using 60% less training data. DeepWalk is also scalable. It is an online learning algorithm which builds useful incremental results, and is trivially parallelizable. These qualities make it suitable for a broad class of real world applications such as network classification, and anomaly detection.

Figures (5)

• Figure 1: Our proposed method learns a latent space representation of social interactions in $\mathbb{R}^d$. The learned representation encodes community structure so it can be easily exploited by standard classification methods. Here, our method is used on Zachary's Karate network zachary1977information to generate a latent representation in $\mathbb{R}^2$. Note the correspondence between community structure in the input graph and the embedding. Vertex colors represent a modularity-based clustering of the input graph.
• Figure 2: The power-law distribution of vertices appearing in short random walks (\ref{['fig:powerlaw-youtube']}) follows a power-law, much like the distribution of words in natural language (\ref{['fig:powerlaw-wiki']}).
• Figure 3: Overview of DeepWalk. We slide a window of length $2w+1$ over the random walk $\mathcal{W}_{v_4}$, mapping the central vertex $v_1$ to its representation $\Phi(v_1)$. Hierarchical Softmax factors out $\Pr(v_3 \mid \Phi(v_1))$ and $\Pr(v_5 \mid \Phi(v_1))$ over sequences of probability distributions corresponding to the paths starting at the root and ending at $v_3$ and $v_5$. The representation $\Phi$ is updated to maximize the probability of $v_1$ co-occurring with its context $\{v_3, v_5\}$.
• Figure 4: Effects of parallelizing DeepWalk
• Figure 5: Parameter Sensitivity Study