Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Permutative redundancy and uncertainty of the objective in deep learning

Vacslav Glukhov

TL;DR

It is shown that traditional architectures are polluted by an astronomical number of equivalent global and local optima and some remedies which reduce or eliminate ghost optima are discussed including forced pre-pruning, re-ordering, ortho-polynomial activations, and modular bio-inspired architectures.

Abstract

Implications of uncertain objective functions and permutative symmetry of traditional deep learning architectures are discussed. It is shown that traditional architectures are polluted by an astronomical number of equivalent global and local optima. Uncertainty of the objective makes local optima unattainable, and, as the size of the network grows, the global optimization landscape likely becomes a tangled web of valleys and ridges. Some remedies which reduce or eliminate ghost optima are discussed including forced pre-pruning, re-ordering, ortho-polynomial activations, and modular bio-inspired architectures.

Permutative redundancy and uncertainty of the objective in deep learning

TL;DR

It is shown that traditional architectures are polluted by an astronomical number of equivalent global and local optima and some remedies which reduce or eliminate ghost optima are discussed including forced pre-pruning, re-ordering, ortho-polynomial activations, and modular bio-inspired architectures.

Abstract

Implications of uncertain objective functions and permutative symmetry of traditional deep learning architectures are discussed. It is shown that traditional architectures are polluted by an astronomical number of equivalent global and local optima. Uncertainty of the objective makes local optima unattainable, and, as the size of the network grows, the global optimization landscape likely becomes a tangled web of valleys and ridges. Some remedies which reduce or eliminate ghost optima are discussed including forced pre-pruning, re-ordering, ortho-polynomial activations, and modular bio-inspired architectures.

Paper Structure

This paper contains 19 sections, 1 theorem, 51 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Uncertain objectives in an uncertain world
  3. Representation of uncertain objective functions
  4. Note on the structure of the optimization objective
  5. Quadractic structure
  6. Gradient convergence near a presumed local quadratic optimum under an uncertain objective
  7. Boltzmann's equilibrium around the presumed optimum
  8. Effects of the anisotropy of the Hessian
  9. Implications of uncertain objectives and permutative redundancy of deep networks
  10. Redundancy of layered structures of permutable elements
  11. Equilibrium stochastic dynamics over multiple redundant optimua
  12. Remedies
  13. Sorensen reordering heuristic
  14. Pre-break the symmetries
  15. Non-permutatiuve pre-pruning
  16. ...and 4 more sections

Key Result

Corollary 1

Let $A = (n_0, \dots, n_d)$ be a neural network architecture. Then any optimum of a loss function over the corresponding parameter space has $\|\pi(A)\|$ redundancies - that is, distinct optima that do not correspond to meaningfully distinct parameterizations - in the loss surface.

Figures (3)

  • Figure 1: Invariance of a two-layer network with respect to the permutation of arbitrary two nodes in a layer. $x$'s are the network's inputs, $y$'s the outputs. Two nodes, $h_{1,2}$ and $h_{1,3}$, of a trained network can be exchanged preserving their inputs and outputs without changing the network's output.
  • Figure 2: Polynomial representation of a set of functions $f^m(x)$ as a one-layer network. Binary pre-pruning (see section \ref{['sec:pre-pruning']} ) ensures non-permutability of the output layer: each $f^m(x)$ represents a different combination of features in the data.
  • Figure 3: Four EMNIST object targets (top) passed through the three bio-inspired spatial filters (middle), located in the feature space defined by the filter intensities (arbitrary units) (bottom, colour indicates different targets)

Theorems & Definitions (2)

  • Corollary
  • Conjecture