Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Robust Contextual Node Representation Learning over Dynamic Graphs

Xingzhi Guo, Silong Wang, Baojian Zhou, Yanghua Xiao, Steven Skiena

TL;DR

A unified dynamic graph learning framework based on sparse node-wise attention is proposed and performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.

Abstract

Real-world graphs grow rapidly with edge and vertex insertions over time, motivating the problem of efficiently maintaining robust node representation over evolving graphs. Recent efficient GNNs are designed to decouple recursive message passing from the learning process, and favor Personalized PageRank (PPR) as the underlying feature propagation mechanism. However, most PPR-based GNNs are designed for static graphs, and efficient PPR maintenance remains as an open problem. Further, there is surprisingly little theoretical justification for the choice of PPR, despite its impressive empirical performance. In this paper, we are inspired by the recent PPR formulation as an explicit $\ell_1$-regularized optimization problem and propose a unified dynamic graph learning framework based on sparse node-wise attention. We also present a set of desired properties to justify the choice of PPR in STOA GNNs, and serves as the guideline for future node attention designs. Meanwhile, we take advantage of the PPR-equivalent optimization formulation and employ the proximal gradient method (ISTA) to improve the efficiency of PPR-based GNNs upto 6 times. Finally, we instantiate a simple-yet-effective model (\textsc{GoPPE}) with robust positional encodings by maximizing PPR previously used as attention. The model performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.

Fast and Robust Contextual Node Representation Learning over Dynamic Graphs

TL;DR

A unified dynamic graph learning framework based on sparse node-wise attention is proposed and performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.

Abstract

Real-world graphs grow rapidly with edge and vertex insertions over time, motivating the problem of efficiently maintaining robust node representation over evolving graphs. Recent efficient GNNs are designed to decouple recursive message passing from the learning process, and favor Personalized PageRank (PPR) as the underlying feature propagation mechanism. However, most PPR-based GNNs are designed for static graphs, and efficient PPR maintenance remains as an open problem. Further, there is surprisingly little theoretical justification for the choice of PPR, despite its impressive empirical performance. In this paper, we are inspired by the recent PPR formulation as an explicit -regularized optimization problem and propose a unified dynamic graph learning framework based on sparse node-wise attention. We also present a set of desired properties to justify the choice of PPR in STOA GNNs, and serves as the guideline for future node attention designs. Meanwhile, we take advantage of the PPR-equivalent optimization formulation and employ the proximal gradient method (ISTA) to improve the efficiency of PPR-based GNNs upto 6 times. Finally, we instantiate a simple-yet-effective model (\textsc{GoPPE}) with robust positional encodings by maximizing PPR previously used as attention. The model performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.

Paper Structure

This paper contains 52 sections, 2 theorems, 11 equations, 2 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Co-evolution Models over Dynamic Graphs
  4. Node Time Series Models over Dynamic Graphs
  5. Maintainable Node Embeddings Models over Dynamic Graphs
  6. Our approach:
  7. Notations and Preliminaries
  8. Notations
  9. Message Passing as Approx. Propagation
  10. PPR as a node proximity design for GNNs
  11. Approx. PPR as $\ell_1$-Regularized Optimization
  12. Algorithmic approaches
  13. Approximate PPR via optimization
  14. Problem Definition
  15. Key Challenges
  16. ...and 37 more sections

Key Result

Lemma 2

Solving PPR in Equ.equ:ppr is equivalent to solving the following $\ell_1$-regularized quadratic objective function where the PPR vector $\bm \pi := \bm D^{1/2} \bm x^{*}$:

Figures (2)

  • Figure 1: A cartoon illustrating contextualized node representation over a dynamic graph. In the beginning, the graph has few edges (black line) and relatively noisy node attributes (various colors). As the graph evolves, the edges and node attributes become more complete, represented by red new edges and more consistent colored attributes. Accordingly, the node-wise attention (the arrows over nodes) for $u_1$ changes, favoring the nodes on its left side.
  • Figure 2: Subfigures (a) and (b) show the CPU time in Major/Minor graph change experiment settings, respectively. The intensity of graph changes between snapshots (major vs minor) may greatly affect the PPR updating time. In both cases, GoPPE is consistently efficient in terms of total CPU time. PPRGo uses Push-S while DynPPE and InsGNN use Push-D. Note that the absolute time is not directly comparable between subfigures due to different settings.

Theorems & Definitions (7)

  • Definition 1: Personalized PageRank
  • Lemma 2: Variational Formulation of Personalized PageRank fountoulakis2019variational
  • Definition 3: Node Classification Problem over Dynamic Graph
  • Theorem 4: PPR Adjustment Rules for Dynamic Graphs
  • Definition 5
  • Definition 6: The Desired Properties of Node Positional Encodings
  • Remark 7