Equivariant neural networks and piecewise linear representation theory
Joel Gibson, Daniel Tubbenhauer, Geordie Williamson
TL;DR
The paper develops a representation-theoretic framework for equivariant neural networks by decomposing layers into simple permutation representations and studying how piecewise-linear activations induce nonlinear maps between these pieces. It reveals a Fourier-like filtration where information flows from low-frequency to high-frequency simple representations, providing a principled lens to interpret network behavior and complexity. A central contribution is the piecewise-linear Schur-type analysis: PL equivariant maps between simple reps are governed by normal subgroups, enabling concrete classifications and rich interaction graphs, especially in cyclic groups. The work combines formal theory with explicit cyclic-group examples (ReLU and Abs), and provides code that allows exploration of more groups, offering a practical toolkit for understanding and debugging equivariant networks.
Abstract
Equivariant neural networks are neural networks with symmetry. Motivated by the theory of group representations, we decompose the layers of an equivariant neural network into simple representations. The nonlinear activation functions lead to interesting nonlinear equivariant maps between simple representations. For example, the rectified linear unit (ReLU) gives rise to piecewise linear maps. We show that these considerations lead to a filtration of equivariant neural networks, generalizing Fourier series. This observation might provide a useful tool for interpreting equivariant neural networks.