Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Guarantees for the DeepWalk Embedding on Block Models

Christopher Harker, Aditya Bhaskara

TL;DR

This work shows convergence properties for the DeepWalk algorithm on graphs obtained from the Stochastic Block Model (SBM), showing that even in the case of one-dimensional embeddings, the output of the DeepWalk algorithm provably recovers the cluster structure with high probability.

Abstract

Graph embeddings have emerged as a powerful tool for understanding the structure of graphs. Unlike classical spectral methods, recent methods such as DeepWalk, Node2Vec, etc. are based on solving nonlinear optimization problems on the graph, using local information obtained by performing random walks. These techniques have empirically been shown to produce ''better'' embeddings than their classical counterparts. However, due to their reliance on solving a nonconvex optimization problem, obtaining theoretical guarantees on the properties of the solution has remained a challenge, even for simple classes of graphs. In this work, we show convergence properties for the DeepWalk algorithm on graphs obtained from the Stochastic Block Model (SBM). Despite being simplistic, the SBM has proved to be a classic model for analyzing the behavior of algorithms on large graphs. Our results mirror the existing ones for spectral embeddings on SBMs, showing that even in the case of one-dimensional embeddings, the output of the DeepWalk algorithm provably recovers the cluster structure with high probability.

Convergence Guarantees for the DeepWalk Embedding on Block Models

TL;DR

This work shows convergence properties for the DeepWalk algorithm on graphs obtained from the Stochastic Block Model (SBM), showing that even in the case of one-dimensional embeddings, the output of the DeepWalk algorithm provably recovers the cluster structure with high probability.

Abstract

Graph embeddings have emerged as a powerful tool for understanding the structure of graphs. Unlike classical spectral methods, recent methods such as DeepWalk, Node2Vec, etc. are based on solving nonlinear optimization problems on the graph, using local information obtained by performing random walks. These techniques have empirically been shown to produce ''better'' embeddings than their classical counterparts. However, due to their reliance on solving a nonconvex optimization problem, obtaining theoretical guarantees on the properties of the solution has remained a challenge, even for simple classes of graphs. In this work, we show convergence properties for the DeepWalk algorithm on graphs obtained from the Stochastic Block Model (SBM). Despite being simplistic, the SBM has proved to be a classic model for analyzing the behavior of algorithms on large graphs. Our results mirror the existing ones for spectral embeddings on SBMs, showing that even in the case of one-dimensional embeddings, the output of the DeepWalk algorithm provably recovers the cluster structure with high probability.

Paper Structure

This paper contains 29 sections, 16 theorems, 97 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Results
  3. Background
  4. Basic Notation
  5. The DeepWalk Algorithm
  6. Constructing the Co-occurrence Matrix.
  7. Computing Embeddings
  8. Stochastic Block Models
  9. Analysis
  10. Co-occurrence Structure and Algorithm
  11. Linear Approximation of Gradient Update
  12. Convergence Analysis
  13. Remark.
  14. Remark.
  15. Experiments
  16. ...and 14 more sections

Key Result

Theorem 1.1

Given a graph $G$ drawn from an SBM with $K$ blocks and parameters $p$ and $q$, DeepWalk embeddings, obtained by initializing a solution in small enough ball $\lVert {\bm{w}}^{(0)} \rVert \le \epsilon$, and training with gradient descent with learning rate $\eta > 0$ small enough, approximately reco

Figures (2)

  • Figure 1: For a random graph drawn from a stochastic block model with $K=3$ blocks and $n=600$, we show the calculated (left) 1-dimensional (middle) 2-dimensional and (right) 3-dimensional embeddings.
  • Figure 2: For random graphs drawn from a stochastic block model with $K=2$ blocks and $n=200, 500, 1000$ nodes, we track the distance between the original nonlinear gradient descent update for deepwalk and its linear approximation.

Theorems & Definitions (24)

  • Theorem 1.1: Informal
  • Lemma 2.1
  • Lemma 3.1
  • Proposition 3.3
  • Lemma 3.4
  • Lemma 3.5
  • Lemma 3.6
  • proof
  • Lemma 3.7
  • proof
  • ...and 14 more