Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Universality of Deep Equivariant Networks

Marco Pacini, Mircea Petrache, Bruno Lepri, Shubhendu Trivedi, Robin Walters

TL;DR

This work shows that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime.

Abstract

Universality results for equivariant neural networks remain rare. Those that do exist typically hold only in restrictive settings: either they rely on regular or higher-order tensor representations, leading to impractically high-dimensional hidden spaces, or they target specialized architectures, often confined to the invariant setting. This work develops a more general account. For invariant networks, we establish a universality theorem under separation constraints, showing that the addition of a fully connected readout layer secures approximation within the class of separation-constrained continuous functions. For equivariant networks, where results are even scarcer, we demonstrate that standard separability notions are inadequate and introduce the sharper criterion of $\textit{entry-wise separability}$. We show that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime. Together with prior results showing the failure of universality for shallow models, our findings identify depth and readout layers as a decisive mechanism for universality, additionally offering a unified perspective that subsumes and extends earlier specialized results.

On Universality of Deep Equivariant Networks

TL;DR

This work shows that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime.

Abstract

Universality results for equivariant neural networks remain rare. Those that do exist typically hold only in restrictive settings: either they rely on regular or higher-order tensor representations, leading to impractically high-dimensional hidden spaces, or they target specialized architectures, often confined to the invariant setting. This work develops a more general account. For invariant networks, we establish a universality theorem under separation constraints, showing that the addition of a fully connected readout layer secures approximation within the class of separation-constrained continuous functions. For equivariant networks, where results are even scarcer, we demonstrate that standard separability notions are inadequate and introduce the sharper criterion of . We show that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime. Together with prior results showing the failure of universality for shallow models, our findings identify depth and readout layers as a decisive mechanism for universality, additionally offering a unified perspective that subsumes and extends earlier specialized results.
Paper Structure (16 sections, 14 theorems, 84 equations)

This paper contains 16 sections, 14 theorems, 84 equations.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Groups and Equivariance
  5. Layer Spaces, Neural Spaces & Equivariant Neural Networks
  6. Universality Classes and Separation
  7. Separation-constrained Universality for Invariant Networks
  8. Universality of Equivariant Neural Networks
  9. Entry-wise Separation
  10. Entry-wise Separation Constrained Universality
  11. Limitations
  12. Conclusions
  13. Separation-constrained Universality for Invariant Networks
  14. Universality of Equivariant Neural Networks
  15. Entry-wise Separation
  16. ...and 1 more sections

Key Result

Theorem 1

Let $M_1, \dots, M_d$ be layer spaces as defined in Definition def:layer-spaces and recall that $I$ denotes the layer space of invariant linear functions from Example example:layer-spaces.example:inv. Set $\rho = \rho\!\bigl(\mathop{\mathrm{\mathcal{U}_\sigma}}\nolimits(M_1, \dots, M_d, I)\bigr)$. T

Theorems & Definitions (38)

  • Definition 1: Layer Spaces
  • Example 1
  • Definition 2: Point-wise Activation
  • Definition 3: Neural Networks and Neural Spaces
  • Example 2
  • Definition 4: Universality Classes
  • Definition 5: Separation-Constrained Universality
  • Theorem 1
  • proof : Proof of Theorem \ref{['th:invariant_universality']}
  • Proposition 1
  • ...and 28 more