Deep Residual Networks and Weight Initialization

TL;DR

This work analyzes why deep residual networks train effectively by examining forward and backward variance propagation under different initializations. It derives a depth-aware weight initialization, Var[w] = c/(nL), to stabilize both activations and gradients, and shows batch normalization can linearize gradient growth, further easing optimization. Empirical results on CIFAR-10 indicate the proposed initialization improves early training stability and can rival batch normalization in some settings. Overall, the paper offers theoretical and empirical insights into initialization and BN strategies to enable truly deep ResNets.

Abstract

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.

Figures (4)

• Figure 1: Residual blocks. Here we use convolutional layer and ReLU for example.
• Figure 2: Learning curves for ResNet with 100 residual blocks. We compared three different setups.
• Figure 3: Nine independent learning curves of ResNet. Weights are initialized by [10].
• Figure 4: Residual block ReLU before convolution.