Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Swapout: Learning an ensemble of deep architectures

Saurabh Singh, Derek Hoiem, David Forsyth

TL;DR

Swapout introduces a general stochastic training framework that unifies dropout, stochastic depth, and residual-style connections by per-unit randomization of outputs, effectively sampling a vast ensemble of architectures during training. This approach yields strong regularization, ties parameters across layers, and enables relatively shallow yet wide networks to match or outperform very deep ResNets on CIFAR-10 and CIFAR-100, with stochastic inference providing additional gains. The work demonstrates that the stochastic schedule and network width significantly influence performance and that Swapout achieves superior parameter efficiency compared to deeper baselines. These findings suggestSwapout as a versatile technique for improving deep networks across architectures, potentially extending to inception, recurrent, and gated networks.

Abstract

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

Swapout: Learning an ensemble of deep architectures

TL;DR

Swapout introduces a general stochastic training framework that unifies dropout, stochastic depth, and residual-style connections by per-unit randomization of outputs, effectively sampling a vast ensemble of architectures during training. This approach yields strong regularization, ties parameters across layers, and enables relatively shallow yet wide networks to match or outperform very deep ResNets on CIFAR-10 and CIFAR-100, with stochastic inference providing additional gains. The work demonstrates that the stochastic schedule and network width significantly influence performance and that Swapout achieves superior parameter efficiency compared to deeper baselines. These findings suggestSwapout as a versatile technique for improving deep networks across architectures, potentially extending to inception, recurrent, and gated networks.

Abstract

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

Paper Structure

This paper contains 23 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Swapout
  4. Notation and terminology:
  5. Inference in Stochastic Networks
  6. Baseline comparison methods
  7. ResNets:
  8. Dropout:
  9. Layer Dropout:
  10. SkipForward:
  11. Experiments
  12. Model:
  13. Training:
  14. Models trained with Swapout consistently outperform baselines:
  15. The stochastic training schedule matters:
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Visualization of architectural differences, showing computations for a block using various architectures. Each circle is a unit in a grid corresponding to spatial layout, and circles are colored to indicate what they report. Given input $X$ ( a), all units in a feed forward block emit $F(X)$ ( b). All units in a residual network block emit $X+F(X)$ ( c). A skipforward network randomly chooses between reporting $X$ and $F(X)$per unit ( d). Finally, swapout randomly chooses between reporting $0$ (and so dropping out the unit), $X$ (skipping the unit), $F(X)$ (imitating a feedforward network at the unit) and $X+F(X)$ (imitating a residual network unit).
  • Figure 2: Stochastic inference needs few samples for a good estimate. We plot the mean error rate on the left as a function of the number of samples for two stochastic training schedules. Standard error of the mean is shown as the shaded interval on the left and magnified in the right plot. It is evident that relatively few samples are needed for a reliable estimate of the mean error. The mean and standard error was computed using 30 repetitions for each sample count.