Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifiable Equivariant Networks are Layerwise Equivariant

Vahid Shahverdi, Giovanni Luca Marchetti, Georg Bökman, Kathlén Kohn

TL;DR

This work proves that end-to-end $G$-equivariance in a deep network implies layerwise $G$-equivariance on latent spaces, provided the model is weakly identifiable. Using an abstract, architecture-agnostic formalism, the authors show that latent actions can be chosen so each layer becomes equivariant, with the overall $G$-action factoring through per-layer symmetry groups. The results apply to a broad class of architectures, including MLPs and multi-head attention, and are supported by illustrative CIFAR-10 experiments that reveal mirror and head-permutation symmetries in learned weights. This theory offers a mathematical explanation for the observed emergence of equivariant structures during training and clarifies when end-to-end symmetry can be realized through layerwise equivariance. The work highlights practical implications for designing equivariant layers and informs the ongoing discussion about learning versus imposing symmetry priors in neural networks.

Abstract

We investigate the relation between end-to-end equivariance and layerwise equivariance in deep neural networks. We prove the following: For a network whose end-to-end function is equivariant with respect to group actions on the input and output spaces, there is a parameter choice yielding the same end-to-end function such that its layers are equivariant with respect to some group actions on the latent spaces. Our result assumes that the parameters of the model are identifiable in an appropriate sense. This identifiability property has been established in the literature for a large class of networks, to which our results apply immediately, while it is conjectural for others. The theory we develop is grounded in an abstract formalism, and is therefore architecture-agnostic. Overall, our results provide a mathematical explanation for the emergence of equivariant structures in the weights of neural networks during training -- a phenomenon that is consistently observed in practice.

Identifiable Equivariant Networks are Layerwise Equivariant

TL;DR

This work proves that end-to-end -equivariance in a deep network implies layerwise -equivariance on latent spaces, provided the model is weakly identifiable. Using an abstract, architecture-agnostic formalism, the authors show that latent actions can be chosen so each layer becomes equivariant, with the overall -action factoring through per-layer symmetry groups. The results apply to a broad class of architectures, including MLPs and multi-head attention, and are supported by illustrative CIFAR-10 experiments that reveal mirror and head-permutation symmetries in learned weights. This theory offers a mathematical explanation for the observed emergence of equivariant structures during training and clarifies when end-to-end symmetry can be realized through layerwise equivariance. The work highlights practical implications for designing equivariant layers and informs the ongoing discussion about learning versus imposing symmetry priors in neural networks.

Abstract

We investigate the relation between end-to-end equivariance and layerwise equivariance in deep neural networks. We prove the following: For a network whose end-to-end function is equivariant with respect to group actions on the input and output spaces, there is a parameter choice yielding the same end-to-end function such that its layers are equivariant with respect to some group actions on the latent spaces. Our result assumes that the parameters of the model are identifiable in an appropriate sense. This identifiability property has been established in the literature for a large class of networks, to which our results apply immediately, while it is conjectural for others. The theory we develop is grounded in an abstract formalism, and is therefore architecture-agnostic. Overall, our results provide a mathematical explanation for the emergence of equivariant structures in the weights of neural networks during training -- a phenomenon that is consistently observed in practice.
Paper Structure (26 sections, 2 theorems, 18 equations, 4 figures)

This paper contains 26 sections, 2 theorems, 18 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Identifiability.
  4. Algebro-geometric and Categorical Deep Learning.
  5. Equivariant Universal Approximation.
  6. Formalism
  7. Models and Submodels
  8. Symmetries and Identifiability
  9. Equivariant Models
  10. Main Result
  11. Examples
  12. Multi-Layer Perceptrons
  13. Symmetries.
  14. Submodels.
  15. Identifiability.
  16. ...and 11 more sections

Key Result

Theorem 4.1

Let $\theta \in \Theta$ be a parameter. Suppose that: Then there exist group actions by $G$ on $V_i$, $i=1, \ldots, L-1$, such that $f_i$ is $G$-equivariant at $\theta_i$, for every $i$.

Figures (4)

  • Figure 1: An image segmentation model is equivariant to rotations of the image. Our main result implies that the group action on the input propagates through the network via latent symmetries (e.g., neuron permutations), until it reaches the output.
  • Figure 2: First-layer weights of MLPs trained on CIFAR10. Each square is a filter that maps an input RGB image to a single neuron of the subsequent layer. The filters have been sorted for illustrative purposes, with different filter categories highlighted by different colors.
  • Figure 3: Visualization of a multi-head attention layer in a neural network autoencoder trained on CIFAR10. The left-most column shows the input image, and the remaining eight columns visualize the attention matrices (for the 'class token') of the heads. The two columns in cyan correspond to attention heads that permute when the input image is left-to-right mirrored.
  • Figure :

Theorems & Definitions (13)

  • Definition 3.1
  • Definition 3.2
  • Remark 3.3
  • Definition 3.4
  • Definition 3.5
  • Definition 3.6
  • Theorem 4.1
  • Remark 4.2
  • Remark 4.3
  • Conjecture 5.1
  • ...and 3 more