Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Any-dimensional equivariant neural networks

Eitan Levin, Mateo Díaz

TL;DR

The paper addresses learning mappings defined on inputs of arbitrary dimension by leveraging representation stability to construct free, dimension-free equivariant neural networks that extend from a fixed training dimension to all dimensions. It shows that equivariant linear layers stabilize across dimensions, enabling finite parameterizations for groups such as permutation, orthogonal, and Lorentz-like actions. To mitigate poor cross-dimension generalization, it introduces a compatibility condition that enforces consistent extensions across dimensions and develops a computational recipe to train networks at a single level and extend them to higher or lower dimensions. Numerical experiments demonstrate improved cross-dimension generalization for compatible networks, and an open-source implementation is provided to facilitate adoption and further research.

Abstract

Traditional supervised learning aims to learn an unknown mapping by fitting a function to a set of input-output pairs with a fixed dimension. The fitted function is then defined on inputs of the same dimension. However, in many settings, the unknown mapping takes inputs in any dimension; examples include graph parameters defined on graphs of any size and physics quantities defined on an arbitrary number of particles. We leverage a newly-discovered phenomenon in algebraic topology, called representation stability, to define equivariant neural networks that can be trained with data in a fixed dimension and then extended to accept inputs in any dimension. Our approach is user-friendly, requiring only the network architecture and the groups for equivariance, and can be combined with any training procedure. We provide a simple open-source implementation of our methods and offer preliminary numerical experiments.

Any-dimensional equivariant neural networks

TL;DR

The paper addresses learning mappings defined on inputs of arbitrary dimension by leveraging representation stability to construct free, dimension-free equivariant neural networks that extend from a fixed training dimension to all dimensions. It shows that equivariant linear layers stabilize across dimensions, enabling finite parameterizations for groups such as permutation, orthogonal, and Lorentz-like actions. To mitigate poor cross-dimension generalization, it introduces a compatibility condition that enforces consistent extensions across dimensions and develops a computational recipe to train networks at a single level and extend them to higher or lower dimensions. Numerical experiments demonstrate improved cross-dimension generalization for compatible networks, and an open-source implementation is provided to facilitate adoption and further research.

Abstract

Traditional supervised learning aims to learn an unknown mapping by fitting a function to a set of input-output pairs with a fixed dimension. The fitted function is then defined on inputs of the same dimension. However, in many settings, the unknown mapping takes inputs in any dimension; examples include graph parameters defined on graphs of any size and physics quantities defined on an arbitrary number of particles. We leverage a newly-discovered phenomenon in algebraic topology, called representation stability, to define equivariant neural networks that can be trained with data in a fixed dimension and then extended to accept inputs in any dimension. Our approach is user-friendly, requiring only the network architecture and the groups for equivariance, and can be combined with any training procedure. We provide a simple open-source implementation of our methods and offer preliminary numerical experiments.
Paper Structure (25 sections, 5 theorems, 14 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 25 sections, 5 theorems, 14 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Core contributions.
  3. Notation.
  4. Outline.
  5. Related work
  6. Equivariant learning.
  7. Dimension-free learning.
  8. Representation stability.
  9. FREE NEURAL NETWORKS
  10. GENERALIZATION ACROSS DIMENSIONS AND COMPATIBILITY
  11. Compatibility in the wild.
  12. Compatible linear layers.
  13. Compatible activation functions.
  14. A COMPUTATIONAL RECIPE FOR LEARNING FREE NEURAL NETWORKS
  15. NUMERICAL EXPERIMENTS
  16. ...and 10 more sections

Key Result

Proposition 2.3

If $\{{\bm V}\}$ is generated in degree $d$, then the orthogonal projections $\mathcal{P}_{{\bm V}}\colon ({\bm V}^+)^{{\bm G}^+}\to {\bm V}^{{\bm G}}$ are injective for all $n\geq d$, and isomorphisms for all large $n$.

Figures (2)

  • Figure 1: Equivariant free neural networks. We use bold font to denote the $n$th element in a sequence; see Notation in Section \ref{['par:notation']} for details.
  • Figure 2: Test errors across dimensions for free and compatible networks. Each experiment is run three times; the lighter bands show the max and min runs, while the bold line shows the average. For diagonal extraction and symmetric projection, we measure the average MSE per entry. Vertical gray lines mark the dimension used for learning.

Theorems & Definitions (18)

  • Definition 2.1: Consistent sequences
  • Definition 2.2: Generation degree
  • Proposition 2.3: Isomorphism of invariants
  • proof
  • Definition 3.1: Compatible networks
  • Theorem 3.2
  • Definition B.1: Stabilizing subgroups
  • Definition B.2: Modules
  • Definition B.3: Induction and algebraically free sequences
  • Example B.4
  • ...and 8 more