Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weisfeiler Leman for Euclidean Equivariant Machine Learning

Snir Hordan, Tal Amir, Nadav Dym

TL;DR

This paper shows that PPGN can simulate $2-WL uniformly on all point clouds with low complexity and extends this result in three ways to prove that a simple modification of this invariant PPGN architecture can be used to obtain a universal equivariant architecture that can approximate all continuous equivariant functions uniformly.

Abstract

The $k$-Weisfeiler-Leman ($k$-WL) graph isomorphism test hierarchy is a common method for assessing the expressive power of graph neural networks (GNNs). Recently, GNNs whose expressive power is equivalent to the $2$-WL test were proven to be universal on weighted graphs which encode $3\mathrm{D}$ point cloud data, yet this result is limited to invariant continuous functions on point clouds. In this paper, we extend this result in three ways: Firstly, we show that PPGN can simulate $2$-WL uniformly on all point clouds with low complexity. Secondly, we show that $2$-WL tests can be extended to point clouds which include both positions and velocities, a scenario often encountered in applications. Finally, we provide a general framework for proving equivariant universality and leverage it to prove that a simple modification of this invariant PPGN architecture can be used to obtain a universal equivariant architecture that can approximate all continuous equivariant functions uniformly. Building on our results, we develop our WeLNet architecture, which sets new state-of-the-art results on the N-Body dynamics task and the GEOM-QM9 molecular conformation generation task.

Weisfeiler Leman for Euclidean Equivariant Machine Learning

TL;DR

This paper shows that PPGN can simulate $2-WL uniformly on all point clouds with low complexity and extends this result in three ways to prove that a simple modification of this invariant PPGN architecture can be used to obtain a universal equivariant architecture that can approximate all continuous equivariant functions uniformly.

Abstract

The -Weisfeiler-Leman (-WL) graph isomorphism test hierarchy is a common method for assessing the expressive power of graph neural networks (GNNs). Recently, GNNs whose expressive power is equivalent to the -WL test were proven to be universal on weighted graphs which encode point cloud data, yet this result is limited to invariant continuous functions on point clouds. In this paper, we extend this result in three ways: Firstly, we show that PPGN can simulate -WL uniformly on all point clouds with low complexity. Secondly, we show that -WL tests can be extended to point clouds which include both positions and velocities, a scenario often encountered in applications. Finally, we provide a general framework for proving equivariant universality and leverage it to prove that a simple modification of this invariant PPGN architecture can be used to obtain a universal equivariant architecture that can approximate all continuous equivariant functions uniformly. Building on our results, we develop our WeLNet architecture, which sets new state-of-the-art results on the N-Body dynamics task and the GEOM-QM9 molecular conformation generation task.
Paper Structure (58 sections, 15 theorems, 85 equations, 1 figure, 5 tables)

This paper contains 58 sections, 15 theorems, 85 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Equivariant Universality
  5. Simulating WL
  6. Problem Setup
  7. Mathematical Notation
  8. 2-WL Tests
  9. Geometric 2-WL Tests
  10. Extensions and Limitations
  11. Simulation of $2$-WL with Exponentially Lower Complexity
  12. PPGN architecture
  13. Analytic PPGN
  14. PPGN separation
  15. Complexity
  16. ...and 43 more sections

Key Result

Theorem 4.1

[2-WL pairwise separation] Let $(n,D,T)\in \mathbb{N}^3$ and set $\Delta=1$. Let $\mathcal{G},\mathcal{G}'\in \mathbb{G}_{n,D}$ represent two graphs separable by $T$ iterations of $2$-WL. Then for Lebesgue almost every choice of the parameters $\theta$, the features $\mathbf{c}_{\mathrm{global}}$ an

Figures (1)

  • Figure 1: Separation results for PPGN on the EXP dataset with a single neuron per node. As our theorem predicts, the Sofplus and leaky ELU activations which are analtyic can separate all graph pairs with a single neuron per node-pair. The separation gap is the norm of the difference between the representations of each pair of graphs, measuring how distinct they are. Non-analytic ReLU and leaky ReLU activations yield consistently diminished separation in comparison with their analytic counterparts.

Theorems & Definitions (32)

  • Theorem 4.1
  • Theorem 4.2
  • proof : Proof idea for Theorem \ref{['thm:pairs']} and Theorem \ref{['thm:uniform']}
  • Theorem 5.1
  • Theorem 6.1
  • proof : Proof overview
  • Theorem 3.1
  • proof : Proof of Theorem \ref{['thm:pairs']}
  • Lemma 3.1
  • Theorem 3.2: Stone–Weierstrass theorem ( compact spaces)
  • ...and 22 more