Highway and Residual Networks learn Unrolled Iterative Estimation

TL;DR

The paper questions the traditional view that deep networks inherently build progressively higher-level abstractions with depth. It introduces unrolled iterative estimation as a framework wherein blocks refine a shared feature representation across a stage, preserving feature identity while enabling deep architectures via stage transitions. The authors derive Residual and Highway networks from this view, provide experimental and visual corroboration, and compare the two architectures across tasks, showing that gating expressiveness and normalization influence performance and regularization. Overall, the work offers a unified theoretical lens to explain resilience to lesioning and reshuffling, while guiding practical design of deep architectures with stage-wise refinement.

Abstract

The past year saw the introduction of new architectures such as Highway networks and Residual networks which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer. In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation -- a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.

Figures (5)

• Figure 1: Illustrating our usage of blocks and stages in Highway and Residual networks.
• Figure 2: (a) A single neural network layer that directly computes the desired representation. (b) The unrolled iterative estimation stage (e.g. from a Residual network) stretches the computation over three layers by first providing a noisy estimate of that representation, but then iteratively refines it over the next to layers. (c) A classic group of three layers can also distribute the computation, but they would produce a new representation at each layer. The iterative estimation stage in (b) can be seen as a middle ground between a single classic neural network layer, (a), and multiple classic layers, (c).
• Figure 3: Experimental corroboration of \ref{['eq:mean']}. The average estimation error -- an empirical estimate of the LHS in \ref{['eq:mean']} -- for each block of each stage (x-axis). It stays close to zero in all stages of a 50-layer ResNet trained on the ILSVRC-2015 dataset. The standard deviation of the estimation error decreases as depth increases in each stage (left to right), indicating iterative refinement of the representations.
• Figure 4: Feature visualization from chu2017visualizing, reproduced with kind permission of the authors. It shows how the response of a single filter (unit) evolves over the three blocks (shown from left to right) of stage 1 in a 50-layer ResNet trained on ImageNet. On the left of each visualization are the top 9 patches from the ImageNet validation set that maximally activated that filter. To the right the corresponding guided backpropagation springenberg2014striving visualizations are shown.
• Figure 5: Comparing 50-layer Highway vs. Residual networks on ILSVRC-2012 classification.