Model Metamers Reveal Invariances in Graph Neural Networks

TL;DR

The paper investigates whether graph neural networks learn human-like invariances by generating metamers—inputs that produce similar internal activations despite structural or feature changes. It introduces a formal metamer framework for GNNs, derives a local metamer dimension from Jacobian rank, and analyzes activation-volume changes under nonlinearities. Through extensive experiments across datasets and six GNN architectures, the study reveals substantial over-invariance, identifies effective mitigation strategies (ELU activations, adversarial training, residuals), and shows how depth, width, and cross-model transfers shape invariance. The findings offer a principled benchmark and practical guidance for aligning GNN representations with human perceptual invariances, highlighting both shared and architecture-specific inductive biases.

Abstract

In recent years, deep neural networks have been extensively employed in perceptual systems to learn representations endowed with invariances, aiming to emulate the invariance mechanisms observed in the human brain. However, studies in the visual and auditory domains have confirmed that significant gaps remain between the invariance properties of artificial neural networks and those of humans. To investigate the invariance behavior within graph neural networks (GNNs), we introduce a model ``metamers'' generation technique. By optimizing input graphs such that their internal node activations match those of a reference graph, we obtain graphs that are equivalent in the model's representation space, yet differ significantly in both structure and node features. Our theoretical analysis focuses on two aspects: the local metamer dimension for a single node and the activation-induced volume change of the metamer manifold. Utilizing this approach, we uncover extreme levels of representational invariance across several classic GNN architectures. Although targeted modifications to model architecture and training strategies can partially mitigate this excessive invariance, they fail to fundamentally bridge the gap to human-like invariance. Finally, we quantify the deviation between metamer graphs and their original counterparts, revealing unique failure modes of current GNNs and providing a complementary benchmark for model evaluation.

Figures (9)

• Figure 1: Overlap between human and model metamers in input space.
• Figure 2: Metamer generation for GNNs.
• Figure 3: Metamers generated from deeper layers of the model become increasingly unrecognizable to humans.
• Figure 4: Example visualizations.
• Figure 5: Structure-metamer evaluation on GCN and ChebNet.