Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding symmetries in deep networks

Vijay Badrinarayanan, Bamdev Mishra, Roberto Cipolla

TL;DR

The paper addresses weight-space symmetries in deep networks that undermine Euclidean SGD optimization, particularly in architectures with batch normalization. It proposes constraining filters to the unit-norm on the oblique manifold, deriving symmetry-invariant Riemannian gradient updates (UN) for W1, W2, and θ, and implementing them efficiently with ManOpt. Empirically, UN updates improve test performance on MNIST, especially in deeper networks, and act as a hyperparameter-free regularizer. The approach is also shown to scale to SegNet for image segmentation (CamVid), suggesting broad practical applicability to large CNNs and real-world vision tasks.

Abstract

Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights. We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold. Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold. Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational efficiency of the weight updates.

Understanding symmetries in deep networks

TL;DR

The paper addresses weight-space symmetries in deep networks that undermine Euclidean SGD optimization, particularly in architectures with batch normalization. It proposes constraining filters to the unit-norm on the oblique manifold, deriving symmetry-invariant Riemannian gradient updates (UN) for W1, W2, and θ, and implementing them efficiently with ManOpt. Empirically, UN updates improve test performance on MNIST, especially in deeper networks, and act as a hyperparameter-free regularizer. The approach is also shown to scale to SegNet for image segmentation (CamVid), suggesting broad practical applicability to large CNNs and real-world vision tasks.

Abstract

Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights. We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold. Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold. Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational efficiency of the weight updates.

Paper Structure

This paper contains 6 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Architecture and symmetry analysis
  3. Resolving symmetry issues using manifold optimization
  4. Experiments and results
  5. Application to image segmentation
  6. Conclusion

Figures (2)

  • Figure 1: ArchBN: a two layer deep architecture for classification with batch normalization BN.
  • Figure 2: SGD with the proposed UN weight updates, shown in Table \ref{['ProposedUpdates']}, for training SegNet SegNetarXiv. The quality of the predictions as compared to the ground truth indicates a successful training of SegNet.