Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Invariance to Node Identifiers in Graph Neural Networks

Maya Bechler-Speicher, Moshe Eliasof, Carola-Bibiane Schonlieb, Ran Gilad-Bachrach, Amir Globerson

TL;DR

This work addresses the tension between the expressive power gained by attaching node IDs to Graph Neural Networks (GNNs) and the need for invariance to ID values to prevent overfitting. Through theoretical analysis, it shows that enforcing ID invariance in every layer does not improve expressiveness beyond ID-free MP-GNNs, while enabling invariance only in the last layer can yield ID-invariant representations with greater expressiveness than the $1$-WL test. Building on these insights, the authors propose ICON, an ID-invariance through contrast regularizer, which encourages identical final-layer embeddings across different random IDs while preserving task performance. Extensive experiments on real-world and synthetic datasets demonstrate that ICON consistently increases ID invariance, often improves generalization and extrapolation, and speeds up training, making ID usage more robust and practical across diverse GNN architectures.

Abstract

Message-Passing Graph Neural Networks (GNNs) are known to have limited expressive power, due to their message passing structure. One mechanism for circumventing this limitation is to add unique node identifiers (IDs), which break the symmetries that underlie the expressivity limitation. In this work, we highlight a key limitation of the ID framework, and propose an approach for addressing it. We begin by observing that the final output of the GNN should clearly not depend on the specific IDs used. We then show that in practice this does not hold, and thus the learned network does not possess this desired structural property. Such invariance to node IDs may be enforced in several ways, and we discuss their theoretical properties. We then propose a novel regularization method that effectively enforces ID invariance to the network. Extensive evaluations on both real-world and synthetic tasks demonstrate that our approach significantly improves ID invariance and, in turn, often boosts generalization performance.

Towards Invariance to Node Identifiers in Graph Neural Networks

TL;DR

This work addresses the tension between the expressive power gained by attaching node IDs to Graph Neural Networks (GNNs) and the need for invariance to ID values to prevent overfitting. Through theoretical analysis, it shows that enforcing ID invariance in every layer does not improve expressiveness beyond ID-free MP-GNNs, while enabling invariance only in the last layer can yield ID-invariant representations with greater expressiveness than the -WL test. Building on these insights, the authors propose ICON, an ID-invariance through contrast regularizer, which encourages identical final-layer embeddings across different random IDs while preserving task performance. Extensive experiments on real-world and synthetic datasets demonstrate that ICON consistently increases ID invariance, often improves generalization and extrapolation, and speeds up training, making ID usage more robust and practical across diverse GNN architectures.

Abstract

Message-Passing Graph Neural Networks (GNNs) are known to have limited expressive power, due to their message passing structure. One mechanism for circumventing this limitation is to add unique node identifiers (IDs), which break the symmetries that underlie the expressivity limitation. In this work, we highlight a key limitation of the ID framework, and propose an approach for addressing it. We begin by observing that the final output of the GNN should clearly not depend on the specific IDs used. We then show that in practice this does not hold, and thus the learned network does not possess this desired structural property. Such invariance to node IDs may be enforced in several ways, and we discuss their theoretical properties. We then propose a novel regularization method that effectively enforces ID invariance to the network. Extensive evaluations on both real-world and synthetic tasks demonstrate that our approach significantly improves ID invariance and, in turn, often boosts generalization performance.

Paper Structure

This paper contains 40 sections, 4 theorems, 6 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Neural Networks
  4. Node Identifiers and Expressiveness in GNNs
  5. Learning Invariances in Neural Networks
  6. GNNs Overfit IDs
  7. Experimental Setting
  8. Datasets
  9. Evaluation Protocol
  10. Results
  11. Theoretical Analysis of Invariance to IDs and Expressivity
  12. Preliminaries
  13. Learning Invariance to IDs
  14. Experimental Evaluation
  15. Real-world Datasets
  16. ...and 25 more sections

Key Result

Theorem 4.2

A function $f$ can be ID-invariant with respect to a set of graphs $S$ and non-ID-invariant with respect to another set of graphs $S'$.

Figures (1)

  • Figure 1: The test accuracy learning curves of RNI and ICON on the EXP and CEXP daatasets. While both methods reach almost perfect accuracy, ICON offers faster convergence.

Theorems & Definitions (5)

  • Definition 4.1: ID-invariant function
  • Theorem 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 7.1