Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Graph Node Embeddings by Smooth Pair Sampling

Konstantin Kutzkov

TL;DR

This work identifies a pronounced skew in positive pair frequencies produced by random-walk based graph embeddings and introduces a beta-smoothed regularization that reweights positive pairs via $T_\beta\#(u,v)^\beta$ with $\beta \in (0,1]$. It proposes SmoothDeepWalk, an efficient on-the-fly sampling algorithm where candidate positive pairs are selected with probability proportional to $\#(u,v)^{\beta-1}$, yielding predictable sampling counts $S_{u,v}=T_\beta\#(u,v)^\beta$ and reducing overreliance on very frequent pairs. The paper provides theoretical analyses showing how smoothing mitigates data-skew, formalizes a generalized objective, and develops a sketch-based method (Frequent) to estimate heavy-pair frequencies at scale, achieving $O(n\,d + b)$ space and $O(M)$ time. Empirical results on eight graphs demonstrate that smoothing improves link prediction on large, sparse graphs and yields robust, sometimes substantial gains in node classification, with recommended defaults for the smoothing parameter $\beta$ and sketch budget $b$. Overall, pair-frequency smoothing offers a scalable, principled regularization for random-walk embeddings, with potential extensions to other embedding frameworks and graph neural networks.

Abstract

Random walk-based node embedding algorithms have attracted a lot of attention due to their scalability and ease of implementation. Previous research has focused on different walk strategies, optimization objectives, and embedding learning models. Inspired by observations on real data, we take a different approach and propose a new regularization technique. More precisely, the frequencies of node pairs generated by the skip-gram model on random walk node sequences follow a highly skewed distribution which causes learning to be dominated by a fraction of the pairs. We address the issue by designing an efficient sampling procedure that generates node pairs according to their {\em smoothed frequency}. Theoretical and experimental results demonstrate the advantages of our approach.

Learning Graph Node Embeddings by Smooth Pair Sampling

TL;DR

This work identifies a pronounced skew in positive pair frequencies produced by random-walk based graph embeddings and introduces a beta-smoothed regularization that reweights positive pairs via with . It proposes SmoothDeepWalk, an efficient on-the-fly sampling algorithm where candidate positive pairs are selected with probability proportional to , yielding predictable sampling counts and reducing overreliance on very frequent pairs. The paper provides theoretical analyses showing how smoothing mitigates data-skew, formalizes a generalized objective, and develops a sketch-based method (Frequent) to estimate heavy-pair frequencies at scale, achieving space and time. Empirical results on eight graphs demonstrate that smoothing improves link prediction on large, sparse graphs and yields robust, sometimes substantial gains in node classification, with recommended defaults for the smoothing parameter and sketch budget . Overall, pair-frequency smoothing offers a scalable, principled regularization for random-walk embeddings, with potential extensions to other embedding frameworks and graph neural networks.

Abstract

Random walk-based node embedding algorithms have attracted a lot of attention due to their scalability and ease of implementation. Previous research has focused on different walk strategies, optimization objectives, and embedding learning models. Inspired by observations on real data, we take a different approach and propose a new regularization technique. More precisely, the frequencies of node pairs generated by the skip-gram model on random walk node sequences follow a highly skewed distribution which causes learning to be dominated by a fraction of the pairs. We address the issue by designing an efficient sampling procedure that generates node pairs according to their {\em smoothed frequency}. Theoretical and experimental results demonstrate the advantages of our approach.
Paper Structure (36 sections, 8 theorems, 27 equations, 20 figures, 8 tables)

This paper contains 36 sections, 8 theorems, 27 equations, 20 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Smooth Pair Sampling
  4. How does smoothing help?
  5. Efficient Pair Frequency Estimation
  6. Related work
  7. Experimental evaluation
  8. Datasets.
  9. Data skew.
  10. Experimental setting.
  11. Computational complexity.
  12. Link prediction.
  13. Default values for $\beta$ and $b$.
  14. Conclusions and future work
  15. Theoretical analysis
  16. ...and 21 more sections

Key Result

Theorem 1

Let $G=(V, E)$ and $D$ be the corresponding random walk corpus. Let $\beta \in (0, 1]$. The following hold for SmoothDeepWalk:

Figures (20)

  • Figure 1: The effect of smoothing the pair frequencies for the Citeseer graph. The gray vertical line shows the transition point after which smoothing leads to more positive samples for the corresponding pairs.
  • Figure 2: Embedding training time (left)
  • Figure 3: Precision@100 scores for link prediction for varying $\beta$ for the Git and Wikipedia graphs.
  • Figure 4: Precision@100 scores for synthetic graphs with low and high clustering coefficients.
  • Figure 5: Macro-F1 scores for node classification on Cora for $\beta=0.75$ when evaluating embeddings for low-degree nodes.
  • ...and 15 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4