A Characterization Theorem for Equivariant Networks with Point-wise Activations
Marco Pacini, Xiaowen Dong, Bruno Lepri, Gabriele Santin
TL;DR
The paper develops a complete characterization of exact equivariance for neural networks with continuous point-wise activations by pairing activation classes $\mathcal{F}$ with representation groups $\mathcal{M}$ and analyzing their duals $\mathcal{M}(\mathcal{F})$ and $\mathcal{F}(\mathcal{M})$, which stabilize after two iterations. It extends the classical finite-group result to non-finite and compact groups, showing that maximal pairs are finite in number and, for compact groups, up to isomorphism any representation is isomorphic to a permutation or signed-permutation representation, enabling wide activation families. The work yields practical corollaries: finite groups admit only permutation representations for non-odd non-affine activations, and disentangled representations are limited to trivial components; rotation-equivariant networks become invariant, while IGNs admit broader admissible spaces via group quotients. The findings illuminate design constraints and opportunities for geometric deep learning, including IGNs and rotation-equivariant models, and provide a principled framework for selecting activations and representations under symmetry constraints. Limitations include confinement to exact finite-dimensional representations and real-valued activations, suggesting directions toward complex-valued extensions and broader activation classes.
Abstract
Equivariant neural networks have shown improved performance, expressiveness and sample complexity on symmetrical domains. But for some specific symmetries, representations, and choice of coordinates, the most common point-wise activations, such as ReLU, are not equivariant, hence they cannot be employed in the design of equivariant neural networks. The theorem we present in this paper describes all possible combinations of finite-dimensional representations, choice of coordinates and point-wise activations to obtain an exactly equivariant layer, generalizing and strengthening existing characterizations. Notable cases of practical relevance are discussed as corollaries. Indeed, we prove that rotation-equivariant networks can only be invariant, as it happens for any network which is equivariant with respect to connected compact groups. Then, we discuss implications of our findings when applied to important instances of exactly equivariant networks. First, we completely characterize permutation equivariant networks such as Invariant Graph Networks with point-wise nonlinearities and their geometric counterparts, highlighting a plethora of models whose expressive power and performance are still unknown. Second, we show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.