Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Two Sides of Redundancy in Graph Neural Networks

Franka Bause, Samir Moustafa, Johannes Langguth, Wilfried N. Gansterer, Nils M. Kriege

TL;DR

A novel aggregation scheme based on neighborhood trees is developed, which allows for controlling redundancy by pruning redundant branches of unfolding trees underlying standard message passing and can improve the accuracy on widely used benchmark datasets.

Abstract

Message passing neural networks iteratively generate node embeddings by aggregating information from neighboring nodes. With increasing depth, information from more distant nodes is included. However, node embeddings may be unable to represent the growing node neighborhoods accurately and the influence of distant nodes may vanish, a problem referred to as oversquashing. Information redundancy in message passing, i.e., the repetitive exchange and encoding of identical information amplifies oversquashing. We develop a novel aggregation scheme based on neighborhood trees, which allows for controlling redundancy by pruning redundant branches of unfolding trees underlying standard message passing. While the regular structure of unfolding trees allows the reuse of intermediate results in a straightforward way, the use of neighborhood trees poses computational challenges. We propose compact representations of neighborhood trees and merge them, exploiting computational redundancy by identifying isomorphic subtrees. From this, node and graph embeddings are computed via a neural architecture inspired by tree canonization techniques. Our method is less susceptible to oversquashing than traditional message passing neural networks and can improve the accuracy on widely used benchmark datasets.

On the Two Sides of Redundancy in Graph Neural Networks

TL;DR

A novel aggregation scheme based on neighborhood trees is developed, which allows for controlling redundancy by pruning redundant branches of unfolding trees underlying standard message passing and can improve the accuracy on widely used benchmark datasets.

Abstract

Message passing neural networks iteratively generate node embeddings by aggregating information from neighboring nodes. With increasing depth, information from more distant nodes is included. However, node embeddings may be unable to represent the growing node neighborhoods accurately and the influence of distant nodes may vanish, a problem referred to as oversquashing. Information redundancy in message passing, i.e., the repetitive exchange and encoding of identical information amplifies oversquashing. We develop a novel aggregation scheme based on neighborhood trees, which allows for controlling redundancy by pruning redundant branches of unfolding trees underlying standard message passing. While the regular structure of unfolding trees allows the reuse of intermediate results in a straightforward way, the use of neighborhood trees poses computational challenges. We propose compact representations of neighborhood trees and merge them, exploiting computational redundancy by identifying isomorphic subtrees. From this, node and graph embeddings are computed via a neural architecture inspired by tree canonization techniques. Our method is less susceptible to oversquashing than traditional message passing neural networks and can improve the accuracy on widely used benchmark datasets.
Paper Structure (32 sections, 2 theorems, 11 equations, 11 figures, 11 tables, 2 algorithms)

This paper contains 32 sections, 2 theorems, 11 equations, 11 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contribution.
  3. Related Work
  4. Preliminaries
  5. Graph theory.
  6. Weisfeiler-Leman and Message Passing Neural Networks.
  7. Non-Redundant Graph Neural Networks
  8. Removing Information Redundancy
  9. GNNs via unfolding tree canonization.
  10. GNNs via neighborhood tree canonization.
  11. Removing Computational Redundancy
  12. Merging trees into a DAG.
  13. Merging unfolding trees.
  14. Merging neighborhood trees.
  15. Non-Redundant Neural Architecture (DAG-MLP)
  16. ...and 17 more sections

Key Result

Proposition 1

Unfolding tree canonization GNNs, as defined in Eq. eq:canongin, are as expressive as GIN, as defined in Eq. eq:gin.

Figures (11)

  • Figure 1: Graph $G$ and its unfolding trees $F_2^v$ for all $v\in V(G)$.
  • Figure 2: Graph $G$ and the unfolding, $0$- and $1$-redundant neighborhood trees of height $2$ of vertex $v$ (vertex in the upper left of $G$).
  • Figure 3: Graph $G$ and its $0$-NTs $T_{2,0}^v$ for all $v\in V(G)$.
  • Figure 4: Computation DAGs for unfolding (\ref{['fig:unfoldingDAG']}) and $0$-NTs (\ref{['fig:ntreeDAG']}) of height $2$ of graph $G$. And edges in the different layers of the merge DAG of $0$-NTs (\ref{['fig:e1']}), (\ref{['fig:e2']}).
  • Figure 5: Two graphs (\ref{['fig:hexagon']}), (\ref{['fig:triangles']}) that cannot be distinguished by unfolding trees, but by $k$-NTs. Figure (\ref{['fig:unfolding']}) shows the unfolding tree $F_3$, which is the same for all vertices of both graphs, while (\ref{['fig:nts']}) shows the $1$-NTs of the vertices in the hexagon (left) and the triangle (right).
  • ...and 6 more figures

Theorems & Definitions (5)

  • Proposition 1
  • Definition 2: $k$-redundant Neighborhood Tree
  • Theorem 3
  • proof
  • Definition 4: Truncated ePath Tree NEURIPS2022_1bd6f176