Activation Functions for "A Feedforward Unitary Equivariant Neural Network"

TL;DR

This short paper generalises three distinct activation functions tailored for a feedforward unitary equivariant neural network to a single functional form, which represents a broad class of functions, maintains unitary equivariance, and offers greater flexibility for the design of equivariant neural networks.

Abstract

In our previous work [Ma and Chan (2023)], we presented a feedforward unitary equivariant neural network. We proposed three distinct activation functions tailored for this network: a softsign function with a small residue, an identity function, and a Leaky ReLU function. While these functions demonstrated the desired equivariance properties, they limited the neural network's architecture. This short paper generalises these activation functions to a single functional form. This functional form represents a broad class of functions, maintains unitary equivariance, and offers greater flexibility for the design of equivariant neural networks.