Optimization Dynamics of Equivariant and Augmented Neural Networks

TL;DR

This analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role in the optimization of neural networks on symmetric data, and compares the strategy of constraining the architecture to be equivariant to that of using data augmentation.

Abstract

We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models.

Figures (7)

• Figure 1: A graphical summary of our framework. The difference between the nominal network and the augmented one lies in the data, the difference between the nominal network and the equivariant one lies in restricting the layers.
• Figure 2: On the left: filter with cross-shaped support. On the right: filter with skew support. Grey indices correspond to non-zero indices. White indices correspond to zeroed-out indices.
• Figure 3: The architecture consists of three convolutional layers with filters $\varphi$ having support as in Figure \ref{['fig:supports']} (left or right), followed by a flattening and then a fully-connected layer.
• Figure 4: Top left sub-figure (gradient descent): distance in 2-norm of layers from $\mathcal{E}$ per epoch of augmented training. On the left of the other sub-figures: distance in 2-norm of layers from $\mathcal{E}$ averaged over the batches per epoch of augmented training. On the right of the other sub-figures: distance in 2-norm of layers from $\mathcal{E}$ during the first epoch of augmented training. The fainter lines are the individual experiments and the thicker lines are the medians over all experiments. The noticeable outliers in faint yellow are most likely due to numerical errors in calculating the non-unitary embedding operator by solving a linear system. Best viewed in color.
• Figure 5: Average pooling with a $2\times 2$ window and a stride of $2$ is equivariant to rotations of square images of even size.

Theorems & Definitions (31)

• Example 2.1
• Example 2.2
• Example 2.3
• Remark 2.4
• Remark 3.1
• Definition 3.2
• Lemma 3.3
• Proposition 3.4
• Lemma 3.5
• proof : Proof of Proposition \ref{['prop:invcomp']}