Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the power of graph neural networks and the role of the activation function

Sammy Khalife, Amitabh Basu

TL;DR

The paper analyzes how the activation function affects the expressivity of Graph Neural Networks (GNNs) under a fixed architecture. It proves that bounded GNNs with piecewise polynomial activations cannot distinguish certain depth-$2$ rooted trees within any fixed number of iterations, while unbounded GNNs can distinguish them in $2$ iterations, linking boundedness to a gap with color refinement. Conversely, when activations are not piecewise polynomial, a single perceptron can distinguish the root vertices of any pair of depth-$2$ nonisomorphic trees in $2$ iterations, illustrating a drastic increase in power. The results clarify how activation choice governs GNN separation power and connect algebraic and transcendental techniques (piecewise-polynomial analysis and Lindemann–Weierstrass) to foundational questions in descriptive complexity and graph isomorphism tasks.

Abstract

In this article we present new results about the expressivity of Graph Neural Networks (GNNs). We prove that for any GNN with piecewise polynomial activations, whose architecture size does not grow with the graph input sizes, there exists a pair of non-isomorphic rooted trees of depth two such that the GNN cannot distinguish their root vertex up to an arbitrary number of iterations. In contrast, it was already known that unbounded GNNs (those whose size is allowed to change with the graph sizes) with piecewise polynomial activations can distinguish these vertices in only two iterations. It was also known prior to our work that with ReLU (piecewise linear) activations, bounded GNNs are weaker than unbounded GNNs [ACI+22]. Our approach adds to this result by extending it to handle any piecewise polynomial activation function, which goes towards answering an open question formulated by [2021, Grohe] more completely. Our second result states that if one allows activations that are not piecewise polynomial, then in two iterations a single neuron perceptron can distinguish the root vertices of any pair of nonisomorphic trees of depth two (our results hold for activations like the sigmoid, hyperbolic tan and others). This shows how the power of graph neural networks can change drastically if one changes the activation function of the neural networks. The proof of this result utilizes the Lindemann-Weierstrauss theorem from transcendental number theory.

On the power of graph neural networks and the role of the activation function

TL;DR

The paper analyzes how the activation function affects the expressivity of Graph Neural Networks (GNNs) under a fixed architecture. It proves that bounded GNNs with piecewise polynomial activations cannot distinguish certain depth- rooted trees within any fixed number of iterations, while unbounded GNNs can distinguish them in iterations, linking boundedness to a gap with color refinement. Conversely, when activations are not piecewise polynomial, a single perceptron can distinguish the root vertices of any pair of depth- nonisomorphic trees in iterations, illustrating a drastic increase in power. The results clarify how activation choice governs GNN separation power and connect algebraic and transcendental techniques (piecewise-polynomial analysis and Lindemann–Weierstrass) to foundational questions in descriptive complexity and graph isomorphism tasks.

Abstract

In this article we present new results about the expressivity of Graph Neural Networks (GNNs). We prove that for any GNN with piecewise polynomial activations, whose architecture size does not grow with the graph input sizes, there exists a pair of non-isomorphic rooted trees of depth two such that the GNN cannot distinguish their root vertex up to an arbitrary number of iterations. In contrast, it was already known that unbounded GNNs (those whose size is allowed to change with the graph sizes) with piecewise polynomial activations can distinguish these vertices in only two iterations. It was also known prior to our work that with ReLU (piecewise linear) activations, bounded GNNs are weaker than unbounded GNNs [ACI+22]. Our approach adds to this result by extending it to handle any piecewise polynomial activation function, which goes towards answering an open question formulated by [2021, Grohe] more completely. Our second result states that if one allows activations that are not piecewise polynomial, then in two iterations a single neuron perceptron can distinguish the root vertices of any pair of nonisomorphic trees of depth two (our results hold for activations like the sigmoid, hyperbolic tan and others). This shows how the power of graph neural networks can change drastically if one changes the activation function of the neural networks. The proof of this result utilizes the Lindemann-Weierstrauss theorem from transcendental number theory.
Paper Structure (6 sections, 10 theorems, 17 equations, 1 figure)

This paper contains 6 sections, 10 theorems, 17 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Formal statement of results
  3. Overview of the proofs
  4. Collision with piecewise polynomial activations
  5. Activations that are not piecewise polynomial
  6. Acknowledgements

Key Result

Theorem 1

grohe2021logicxu2018powerfulmorris2019weisfeiler Let $d \geq 1$, and let $\xi^{d}$ be an embedding computed by a GNN after $d$ iterations. Then $\mathsf{cr}^{d}$ refines $\xi$, that is, for all graphs $G, G'$ and vertices $v \in V(G)$, $v' \in V(G')$, $\mathsf{cr}^{d}(v)=\mathsf{cr}^{d}(v') \implies

Figures (1)

  • Figure 1: $T[k_1, \cdots, k_m]$

Theorems & Definitions (26)

  • Definition 1: Piecewise polynomial
  • Definition 2: Finitely generated polynomial
  • Definition 3: Embedding, equivariance, and refinement
  • Definition 4: Color refinement
  • Remark 1
  • Definition 5: Graph Neural Network (GNN)
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • ...and 16 more